Oct 30 13:48:18.102510 ip-10-0-131-200 systemd[1]: Starting Kubernetes Kubelet... Oct 30 13:48:18.483480 ip-10-0-131-200 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:18.483480 ip-10-0-131-200 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 13:48:18.483480 ip-10-0-131-200 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:18.483480 ip-10-0-131-200 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:48:18.487533 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487449 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:18.487533 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487533 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:18.487533 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487537 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487541 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487543 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487546 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487549 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487552 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487555 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487557 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487560 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487563 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487565 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487568 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487571 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487573 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487576 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487578 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487581 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487583 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487587 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:18.487635 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487589 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487592 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487595 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487599 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487601 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487603 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487606 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487608 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487611 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487613 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487616 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487618 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487621 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487623 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487627 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487629 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487632 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487635 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487638 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487641 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:18.488070 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487644 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487647 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487650 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487655 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487659 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487664 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487668 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487671 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487674 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487677 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487680 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487684 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487686 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487689 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487692 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487694 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487697 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487699 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:18.488565 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487702 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487722 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487726 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487729 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487731 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487734 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487737 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487740 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487743 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487746 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487749 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487751 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487754 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487757 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487760 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487762 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487765 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487768 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487772 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487774 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:18.489014 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487777 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487780 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487782 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487785 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487788 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487792 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487796 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487798 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487801 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487803 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487806 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487809 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487812 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487814 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487817 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487819 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487822 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487824 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487827 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:18.489536 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487830 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487834 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487837 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487840 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487842 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487845 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.487848 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488642 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488650 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488653 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488656 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488659 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488662 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488665 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488667 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488670 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488673 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488676 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488679 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488682 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:18.490401 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488685 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488687 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488690 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488692 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488695 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488698 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488700 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488704 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488706 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488709 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488711 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488714 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488717 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488720 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488722 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488725 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488728 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488730 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488735 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:18.490936 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488739 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488743 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488747 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488750 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488753 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488755 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488758 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488760 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488763 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488765 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488768 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488770 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488773 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488776 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488778 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488781 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488783 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488787 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488790 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:18.491411 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488792 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488795 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488797 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488800 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488803 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488805 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488809 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488811 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488814 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488817 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488819 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488822 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488825 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488827 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488830 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488832 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488835 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488837 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488840 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488842 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:18.491877 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488845 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488847 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488850 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488852 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488855 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488858 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488861 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488863 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488866 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488869 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488871 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488874 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488876 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488878 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488883 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488886 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488889 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488892 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:18.492374 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488894 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488897 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488899 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488902 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488905 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488907 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488910 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488913 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488915 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488918 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488920 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488924 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488926 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488929 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488931 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.488934 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489838 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489852 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489858 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489865 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:18.492818 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489871 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489876 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489881 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489885 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489896 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489900 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489905 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489909 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489915 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489921 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489925 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489929 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489934 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489938 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489942 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489946 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489955 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489959 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489964 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489968 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:18.493319 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489972 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489977 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489981 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489985 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489989 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489994 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.489998 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490002 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490007 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490018 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490023 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490029 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490035 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490040 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490044 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490050 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490057 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490063 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490068 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:18.493826 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490072 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490077 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490086 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490091 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490096 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490100 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490105 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490110 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490115 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490119 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490123 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490129 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490134 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490138 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490147 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490152 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490156 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490161 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490165 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:18.494299 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490169 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490173 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490177 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490181 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490207 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490212 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490216 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490221 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490231 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490236 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490240 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490245 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490249 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490253 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490258 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490263 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490267 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490272 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490276 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490281 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:18.494766 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490290 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490294 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490298 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490303 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490307 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490311 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490315 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490318 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490322 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490326 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490330 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490334 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490339 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490347 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490353 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490423 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490445 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490559 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490566 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490576 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:18.495293 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490580 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490583 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490585 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.490589 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491457 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491468 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491476 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491481 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491486 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491490 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491495 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491499 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491503 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491506 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491510 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491513 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491516 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491519 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491522 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491525 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491528 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491531 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491536 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491540 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 13:48:18.495762 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491543 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491546 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491550 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491554 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491558 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491561 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491565 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491568 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491571 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491574 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491577 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491580 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491585 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491588 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491591 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491594 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491597 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491600 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491606 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491609 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491613 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491616 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491619 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491624 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 13:48:18.496334 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491629 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491633 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491638 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491643 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491647 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491652 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491656 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491660 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491664 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491668 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491675 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491684 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491687 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491691 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491694 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491697 2579 flags.go:64] FLAG: --help="false" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491700 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491703 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491706 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491709 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491712 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491716 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491719 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 13:48:18.496954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491722 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491725 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491728 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491731 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491734 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491737 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491740 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491743 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491746 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491748 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491751 2579 flags.go:64] FLAG: --lock-file="" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491754 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491757 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491760 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491766 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491769 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491772 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491775 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491778 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491782 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491787 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491790 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491795 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491798 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491802 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 13:48:18.497537 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491805 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491808 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491811 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491814 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491817 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491820 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491823 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491832 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491835 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491838 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491841 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491844 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491847 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491850 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491853 2579 flags.go:64] FLAG: --port="10250" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491856 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491858 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f6456afe4bcccee" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491862 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491865 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491868 2579 flags.go:64] FLAG: --register-node="true" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491872 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491876 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491879 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491882 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491885 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491889 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 13:48:18.498268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491892 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491897 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491900 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491903 2579 flags.go:64] FLAG: --runonce="false" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491906 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491910 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491913 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491915 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491918 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491921 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491924 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491927 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491930 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491933 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491936 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491939 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491942 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491945 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491947 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491954 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491957 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491960 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491964 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491967 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491970 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 13:48:18.498911 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491973 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491976 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491980 2579 flags.go:64] FLAG: --v="2" Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491984 2579 flags.go:64] FLAG: --version="false" Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491988 2579 flags.go:64] FLAG: --vmodule="" Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491992 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.491995 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492155 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492161 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492165 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492169 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492172 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492175 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492178 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492181 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492183 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492198 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492201 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492204 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492207 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:18.499506 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492209 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492212 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492215 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492218 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492220 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492223 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492226 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492228 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492231 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492234 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492236 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492239 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492242 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492245 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492248 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492250 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492253 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492255 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492258 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492260 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:18.499983 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492264 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492267 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492270 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492272 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492275 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492278 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492281 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492284 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492286 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492289 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492292 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492295 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492297 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492300 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492302 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492305 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492307 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492310 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492312 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492315 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:18.500521 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492317 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492320 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492323 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492326 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492328 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492331 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492335 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492339 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492342 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492345 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492348 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492350 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492354 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492357 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492360 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492362 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492365 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492367 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492370 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492373 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:18.500987 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492375 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492380 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492383 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492387 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492389 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492392 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492395 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492397 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492400 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492402 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492405 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492407 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492410 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492412 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492415 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492417 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492420 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492423 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492426 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:18.501486 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492428 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492431 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492434 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492436 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492439 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492444 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492447 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492449 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492453 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492455 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492458 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492460 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492463 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492466 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.492494 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492535 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:18.501934 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492540 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492544 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492547 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492550 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492553 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492555 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492558 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492561 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492564 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492566 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492569 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492572 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492575 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492577 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492580 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492582 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492585 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492589 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492593 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492596 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:18.502344 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492599 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492603 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492607 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492610 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492612 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492615 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492618 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492620 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492623 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492626 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492629 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492631 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492635 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492640 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492643 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492645 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492649 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492652 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492655 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:18.502814 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492657 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492660 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492662 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492665 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492667 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492670 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492673 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492675 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492678 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492680 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492683 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492686 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492689 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492692 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492696 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492700 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492702 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492705 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492708 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:18.503301 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492711 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492714 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492716 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492719 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492722 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492724 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492727 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492729 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492732 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492735 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492737 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492740 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492743 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492745 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492748 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492750 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492753 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492755 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492758 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:18.503785 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492760 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492763 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492765 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492768 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492770 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492773 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492776 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492778 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492782 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492786 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492789 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492791 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492794 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492796 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492799 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492801 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492804 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492807 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492809 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492811 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:18.504262 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492814 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492816 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492819 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492822 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492825 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492829 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492832 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.492834 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.499800 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.499819 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.499837 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.499842 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499904 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499909 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499912 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499916 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499918 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499921 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499924 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:18.504735 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499927 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499930 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499933 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499935 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499938 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499941 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499945 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499950 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499953 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499956 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499959 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499962 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499965 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499968 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499970 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499973 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499975 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499978 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499981 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:18.505175 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499983 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499986 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499990 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499994 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.499997 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500000 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500003 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500006 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500009 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500011 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500014 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500016 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500019 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500021 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500024 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500027 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500030 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500032 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500035 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500038 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:18.505685 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500041 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500043 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500046 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500048 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500051 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500054 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500056 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500059 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500062 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500064 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500067 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500070 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500072 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500076 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500080 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500084 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500092 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500095 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500098 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:18.506166 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500101 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500103 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500106 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500109 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500112 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500114 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500117 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500120 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500123 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500125 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500128 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500131 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500133 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500136 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500139 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500141 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500144 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500147 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500150 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500153 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:18.506652 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500155 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500157 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500160 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500163 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500165 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500168 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500171 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500174 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500177 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500180 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500183 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500201 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500204 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500207 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500210 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500212 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500215 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500218 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500220 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:18.507135 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500223 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500226 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.500255 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500406 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500411 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500414 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500417 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500420 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500423 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500426 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500429 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500432 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500437 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500440 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500444 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500447 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:18.507708 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500450 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500453 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500456 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500459 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500462 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500465 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500468 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500471 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500475 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500478 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500480 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500483 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500485 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500489 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500491 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500493 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500496 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500498 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500501 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500504 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:18.508114 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500506 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500509 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500511 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500514 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500517 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500519 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500522 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500525 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500528 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500532 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500534 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500537 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500540 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500542 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500545 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500548 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500550 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500553 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500555 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:18.508612 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500558 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500561 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500564 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500567 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500569 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500572 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500575 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500577 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500580 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500582 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500585 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500588 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500591 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500593 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500596 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500598 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500601 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500604 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500606 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500609 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:18.509169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500612 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500615 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500617 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500620 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500622 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500625 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500628 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500630 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500633 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500635 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500638 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500641 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500644 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500646 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500649 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500652 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500654 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500657 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500659 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:18.509712 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500662 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500664 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500667 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500670 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500673 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500675 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500678 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500680 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500683 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500686 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500688 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500691 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500693 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500696 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500699 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.500721 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:48:18.510159 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500756 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500760 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500763 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500766 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500769 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500772 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500774 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500777 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500780 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500782 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500785 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500787 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500790 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500794 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500796 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500799 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500802 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500804 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500807 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:48:18.510568 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500810 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500812 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500815 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500817 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500820 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500823 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500825 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500828 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500830 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500833 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500836 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500838 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500841 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500843 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500846 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500849 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500851 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500854 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500857 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500860 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:48:18.511018 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500863 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500865 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500868 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500870 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500873 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500875 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500878 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500881 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500884 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500886 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500889 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500891 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500894 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500896 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500899 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500901 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500904 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500906 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500909 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:48:18.511608 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500911 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500914 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500916 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500919 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500921 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500924 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500926 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500929 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500931 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500934 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500936 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500941 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500944 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500947 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500949 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500952 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500955 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500957 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500960 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:48:18.512050 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500963 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500965 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500969 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500971 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500974 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500976 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500980 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500982 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500985 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500988 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500990 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500993 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500996 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.500999 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501003 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501006 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501009 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501013 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501017 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:48:18.512549 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501020 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501023 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501025 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501028 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501031 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501035 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501038 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501041 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501044 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:18.501046 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.501111 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.503487 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.504244 2579 server.go:1014] "Starting client certificate rotation" Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.504333 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.504368 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:48:18.513000 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.506226 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:48:18.516674 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.516655 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:48:18.528472 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.528451 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 13:48:18.531804 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.531790 2579 log.go:25] "Validated CRI v1 image API" Oct 30 13:48:18.532675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.532648 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 13:48:18.536548 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.536527 2579 fs.go:135] Filesystem UUIDs: map[157e2c80-860f-4a35-be8e-94ada3792988:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b1a33dc4-1384-4315-9baf-386c7ee93c21:/dev/nvme0n1p4] Oct 30 13:48:18.536613 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.536547 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 13:48:18.541591 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.541468 2579 manager.go:217] Machine: {Timestamp:2026-10-30 13:48:18.540365312 +0000 UTC m=+0.326723352 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109811 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c2f910603bf86ed635fc1f92fcb3 SystemUUID:ec28c2f9-1060-3bf8-6ed6-35fc1f92fcb3 BootID:98b49e1e-8273-4d80-9a06-931cfc104110 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c7:4a:95:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c7:4a:95:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:17:f7:19:b9:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 13:48:18.542213 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.542200 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 13:48:18.542315 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.542302 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 13:48:18.544540 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.544513 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 13:48:18.544681 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.544541 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 13:48:18.544723 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.544684 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 13:48:18.544723 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.544692 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 13:48:18.544723 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.544721 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:48:18.546204 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.546178 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:48:18.547303 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.547291 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 13:48:18.551289 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.551275 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 13:48:18.551427 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.551417 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 13:48:18.552129 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.552120 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 13:48:18.552163 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.552134 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 13:48:18.552163 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.552145 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 13:48:18.552163 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.552154 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 13:48:18.552163 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.552163 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 13:48:18.552557 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.552541 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:48:18.554878 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.554850 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 13:48:18.560754 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.560739 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:48:18.561444 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.561431 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 13:48:18.561490 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.561462 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 13:48:18.561631 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.561621 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 13:48:18.561663 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.561635 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 13:48:18.561663 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.561641 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 13:48:18.561663 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.561647 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 13:48:18.561663 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.561652 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 13:48:18.561663 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.561659 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 13:48:18.562065 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.562057 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 13:48:18.562094 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.562066 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 13:48:18.562094 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.562073 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 13:48:18.562094 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.562079 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 13:48:18.562094 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.562088 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 13:48:18.562263 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.562097 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 13:48:18.562860 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.562851 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 13:48:18.562890 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.562861 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 13:48:18.566582 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.566569 2579 server.go:1263] "Started kubelet" Oct 30 13:48:18.566686 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.566659 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 13:48:18.566777 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.566737 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 13:48:18.566824 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.566797 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 13:48:18.568353 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.568336 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 13:48:18.568647 ip-10-0-131-200 systemd[1]: Started Kubernetes Kubelet. Oct 30 13:48:18.572158 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.572140 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 13:48:18.573657 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.573639 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 13:48:18.576689 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.576670 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 13:48:18.577724 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.577704 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 13:48:18.577857 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.577833 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:18.578433 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.578416 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 13:48:18.578544 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.578469 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 13:48:18.578544 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.578481 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 13:48:18.578661 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.578603 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 13:48:18.578661 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.578612 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 13:48:18.579085 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.579067 2579 factory.go:153] Registering CRI-O factory Oct 30 13:48:18.579085 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.579086 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 13:48:18.579246 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.579133 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 13:48:18.579246 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.579140 2579 factory.go:55] Registering systemd factory Oct 30 13:48:18.579246 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.579145 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 13:48:18.579246 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.579157 2579 factory.go:103] Registering Raw factory Oct 30 13:48:18.579246 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.579168 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 13:48:18.580341 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.580315 2579 manager.go:319] Starting recovery of all containers Oct 30 13:48:18.592250 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.589665 2579 manager.go:324] Recovery completed Oct 30 13:48:18.594717 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.594700 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:18.596682 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.596650 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:18.596742 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.596702 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:18.596742 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.596721 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:18.598095 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.598020 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 13:48:18.598095 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.598020 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 13:48:18.599023 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.598976 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 13:48:18.599023 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.599020 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 13:48:18.599150 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.599039 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 13:48:18.599370 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.599353 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:18.600558 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.600532 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:18.600653 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.600570 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:18.600653 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.600591 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:18.603268 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.603251 2579 policy_none.go:50] "Start" Oct 30 13:48:18.603332 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.603271 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 13:48:18.603332 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.603284 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 13:48:18.605255 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.605244 2579 policy_none.go:44] "Start" Oct 30 13:48:18.607422 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.606652 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-200.ec2.internal.18e352c8b2938932 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-200.ec2.internal,},FirstTimestamp:2026-10-30 13:48:18.56653957 +0000 UTC m=+0.352897610,LastTimestamp:2026-10-30 13:48:18.56653957 +0000 UTC m=+0.352897610,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-200.ec2.internal,}" Oct 30 13:48:18.609816 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.609796 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 13:48:18.614452 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.614422 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jwmt4" Oct 30 13:48:18.615129 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.615107 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 13:48:18.615174 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.615107 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 13:48:18.628309 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.628287 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jwmt4" Oct 30 13:48:18.641281 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.641255 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 13:48:18.641384 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.641217 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-200.ec2.internal.18e352c8b45f7c0d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-200.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-200.ec2.internal,},FirstTimestamp:2026-10-30 13:48:18.596682765 +0000 UTC m=+0.383040815,LastTimestamp:2026-10-30 13:48:18.596682765 +0000 UTC m=+0.383040815,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-200.ec2.internal,}" Oct 30 13:48:18.649975 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.649945 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 13:48:18.652622 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.652608 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 13:48:18.652716 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.652638 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 13:48:18.652716 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.652647 2579 server.go:87] "Starting device plugin registration server" Oct 30 13:48:18.652954 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.652941 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 13:48:18.652999 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.652956 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 13:48:18.653062 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.653045 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 13:48:18.653158 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.653124 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 13:48:18.653158 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.653134 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 13:48:18.653809 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.653789 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 13:48:18.653885 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.653844 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:18.720289 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.720246 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 13:48:18.722126 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.721506 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 13:48:18.722126 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.721526 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 13:48:18.722126 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.721550 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 13:48:18.722126 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.721589 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 13:48:18.740982 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.740916 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:18.753634 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.753615 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:18.754655 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.754636 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:18.754737 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.754670 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:18.754737 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.754680 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:18.754852 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.754707 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.777340 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.777316 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.777447 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.777346 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-200.ec2.internal\": node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:18.820975 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.820947 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:18.821967 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.821949 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal"] Oct 30 13:48:18.822043 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.822018 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:18.822932 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.822916 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:18.823031 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.822959 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:18.823031 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.822974 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:18.825175 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.825159 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:18.825346 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.825330 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.825394 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.825362 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:18.825935 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.825918 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:18.826008 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.825952 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:18.826008 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.825966 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:18.826008 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.825919 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:18.826008 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.825998 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:18.826008 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.826010 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:18.827948 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.827930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.828041 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.827955 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:18.828671 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.828657 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:18.828723 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.828685 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:18.828723 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.828696 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:18.855795 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.855771 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-200.ec2.internal\" not found" node="ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.858449 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.858433 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-200.ec2.internal\" not found" node="ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.921267 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:18.921238 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:18.979336 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.979305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.979421 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.979339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:18.979421 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:18.979362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bda9b1f2dbe238b1af8586cc49b37f56-config\") pod \"kube-apiserver-proxy-ip-10-0-131-200.ec2.internal\" (UID: \"bda9b1f2dbe238b1af8586cc49b37f56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.021379 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:19.021310 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:19.080131 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.080100 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.080131 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.080107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.080257 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.080152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.080257 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.080169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bda9b1f2dbe238b1af8586cc49b37f56-config\") pod \"kube-apiserver-proxy-ip-10-0-131-200.ec2.internal\" (UID: \"bda9b1f2dbe238b1af8586cc49b37f56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.080257 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.080217 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bda9b1f2dbe238b1af8586cc49b37f56-config\") pod \"kube-apiserver-proxy-ip-10-0-131-200.ec2.internal\" (UID: \"bda9b1f2dbe238b1af8586cc49b37f56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.080343 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.080257 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.121574 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:19.121542 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:19.160327 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.160297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.162454 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.162432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.222206 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:19.222147 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:19.322683 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:19.322615 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:19.422846 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:19.422797 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:19.504182 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.504131 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 13:48:19.504707 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.504322 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 13:48:19.523280 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:19.523244 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:19.624361 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:19.624276 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:19.633687 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.633622 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 13:43:19 +0000 UTC" deadline="2028-06-28 11:15:21.925239857 +0000 UTC" Oct 30 13:48:19.633687 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.633676 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14565h27m2.291567352s" Oct 30 13:48:19.658588 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.658560 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:48:19.659655 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.659628 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:48:19.659781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.659667 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:48:19.659781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.659681 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:48:19.724373 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:19.724337 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:48:19.771714 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.771683 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:19.771872 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.771782 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:19.778019 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.777984 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.804927 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.804896 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:48:19.805982 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.805967 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:48:19.811783 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.811758 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:19.831538 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.831506 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:48:19.850006 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:19.849961 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda9b1f2dbe238b1af8586cc49b37f56.slice/crio-171b1fcd87d07a1dd74252030a4a51cb0af23e11ba6bdd5b097fd6fd086c634a WatchSource:0}: Error finding container 171b1fcd87d07a1dd74252030a4a51cb0af23e11ba6bdd5b097fd6fd086c634a: Status 404 returned error can't find the container with id 171b1fcd87d07a1dd74252030a4a51cb0af23e11ba6bdd5b097fd6fd086c634a Oct 30 13:48:19.854638 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:19.854589 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950f8228a09238446a40b1b7667155e5.slice/crio-b7aac1d6b69598fa39e2a64d5614ca1c7e2f159028be83206d735501d5477508 WatchSource:0}: Error finding container b7aac1d6b69598fa39e2a64d5614ca1c7e2f159028be83206d735501d5477508: Status 404 returned error can't find the container with id b7aac1d6b69598fa39e2a64d5614ca1c7e2f159028be83206d735501d5477508 Oct 30 13:48:19.854974 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:19.854957 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:48:20.514678 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.514643 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:48:20.553222 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.553175 2579 apiserver.go:52] "Watching apiserver" Oct 30 13:48:20.578234 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.578185 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 13:48:20.603002 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.602968 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 13:48:20.603442 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.603411 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-stdsc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal","openshift-network-operator/iptables-alerter-tghzm","openshift-dns/node-resolver-p7256","openshift-multus/multus-additional-cni-plugins-6wf4d","openshift-multus/network-metrics-daemon-gwvzz","openshift-network-diagnostics/network-check-target-ldf6b","kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8b7mh","openshift-multus/multus-qb4l7","openshift-image-registry/node-ca-pscjw","openshift-ovn-kubernetes/ovnkube-node-2l4sz","kube-system/global-pull-secret-syncer-pb5tm"] Oct 30 13:48:20.606592 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.606564 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:20.606717 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.606700 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:20.608824 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.608802 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.611475 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.611449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.611608 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.611532 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:20.613934 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.613908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.616927 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.616900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.620434 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.620406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.623324 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.623303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:20.623418 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.623378 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:20.626914 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.626888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:20.628923 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.628896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.631558 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.631531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.634224 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.634109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.636717 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.636692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.637129 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637108 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 13:48:20.637258 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637234 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:20.637456 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637353 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:20.637456 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637439 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 13:48:20.637580 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637533 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 13:48:20.637580 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637575 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 13:48:20.637672 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637615 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 13:48:20.637707 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637698 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 13:48:20.637853 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637788 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2dbfk\"" Oct 30 13:48:20.638527 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.637879 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.639319 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5pv6\"" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.639362 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5r2q6\"" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.639425 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.640076 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.640528 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-697m8\"" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.640535 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.640623 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.640677 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 13:48:20.640793 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.640760 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rlln7" Oct 30 13:48:20.653513 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.653483 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rlln7" Oct 30 13:48:20.680012 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.679972 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 13:48:20.690595 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-tmp\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.690671 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-socket-dir-parent\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.690671 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-kubelet\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.690743 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-log-socket\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.690743 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-var-lib-kubelet\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.690743 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtsfg\" (UniqueName: \"kubernetes.io/projected/72a2c31f-6357-4b42-be58-9672ec8dcecb-kube-api-access-dtsfg\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.690841 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.690841 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-run\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.690841 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-sys\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.690841 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mshk\" (UniqueName: \"kubernetes.io/projected/0f87e904-4c9c-4691-8858-dc3dfbe443af-kube-api-access-9mshk\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a2c31f-6357-4b42-be58-9672ec8dcecb-tmp-dir\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-registration-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-system-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-var-lib-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.690978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72a2c31f-6357-4b42-be58-9672ec8dcecb-hosts-file\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4wd\" (UniqueName: \"kubernetes.io/projected/a53bb1dd-634e-4c2f-9e71-306c2136c837-kube-api-access-4h4wd\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-script-lib\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.691188 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-system-cni-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2d8\" (UniqueName: \"kubernetes.io/projected/a40f373a-ac17-46aa-affb-41fc629af3da-kube-api-access-ft2d8\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-daemon-config\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-kubernetes\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-netns\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gjnm\" (UniqueName: \"kubernetes.io/projected/f30df3c3-736e-4aad-bd2f-44a8866b3f93-kube-api-access-4gjnm\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-cni-binary-copy\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-bin\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-kubelet-config\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysconfig\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.691781 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-systemd-units\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-ovn\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-env-overrides\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-host\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.691987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a40f373a-ac17-46aa-affb-41fc629af3da-iptables-alerter-script\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-conf-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-node-log\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-cnibin\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-modprobe-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-conf\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-socket-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-netd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggnrl\" (UniqueName: \"kubernetes.io/projected/c59f3642-4311-4da7-b3e2-66c782e2b83b-kube-api-access-ggnrl\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.692596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-k8s-cni-cncf-io\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-bin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-etc-kubernetes\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e9025a0-9daa-426a-b18d-3a04a154b693-host\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e9025a0-9daa-426a-b18d-3a04a154b693-serviceca\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-config\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-systemd\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-sys-fs\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjs8f\" (UniqueName: \"kubernetes.io/projected/7e9025a0-9daa-426a-b18d-3a04a154b693-kube-api-access-qjs8f\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-slash\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-dbus\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-lib-modules\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-tuned\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40f373a-ac17-46aa-affb-41fc629af3da-host-slash\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-device-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.693620 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-multus-certs\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpq4l\" (UniqueName: \"kubernetes.io/projected/67379085-0be6-451f-b409-4e47dd190877-kube-api-access-dpq4l\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.692987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-systemd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-cnibin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-multus\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-kubelet\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-netns\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-etc-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-hostroot\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhn2\" (UniqueName: \"kubernetes.io/projected/f11ccc81-ad15-4901-afe8-dbc49587fb90-kube-api-access-9xhn2\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-agent-certs\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:20.694285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.693352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-konnectivity-ca\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:20.728446 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.728383 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" event={"ID":"950f8228a09238446a40b1b7667155e5","Type":"ContainerStarted","Data":"b7aac1d6b69598fa39e2a64d5614ca1c7e2f159028be83206d735501d5477508"} Oct 30 13:48:20.729665 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.729600 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" event={"ID":"bda9b1f2dbe238b1af8586cc49b37f56","Type":"ContainerStarted","Data":"171b1fcd87d07a1dd74252030a4a51cb0af23e11ba6bdd5b097fd6fd086c634a"} Oct 30 13:48:20.741897 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.741866 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 13:48:20.742109 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.742074 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 13:48:20.742185 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.742157 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 13:48:20.742282 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.742184 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 13:48:20.742282 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.742270 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:48:20.742894 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.742869 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 13:48:20.743008 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.742970 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jc894\"" Oct 30 13:48:20.743073 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743015 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 13:48:20.743118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743082 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 13:48:20.743165 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743118 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 13:48:20.743198 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743165 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 13:48:20.743251 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743219 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 13:48:20.743305 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743299 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mxshj\"" Oct 30 13:48:20.743399 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743385 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 13:48:20.743472 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743461 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xn5x9\"" Oct 30 13:48:20.743512 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743491 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 13:48:20.743569 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743548 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7j7wh\"" Oct 30 13:48:20.743624 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743593 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 13:48:20.743680 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.743665 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k4gxw\"" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-log-socket\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-var-lib-kubelet\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsfg\" (UniqueName: \"kubernetes.io/projected/72a2c31f-6357-4b42-be58-9672ec8dcecb-kube-api-access-dtsfg\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-run\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-sys\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mshk\" (UniqueName: \"kubernetes.io/projected/0f87e904-4c9c-4691-8858-dc3dfbe443af-kube-api-access-9mshk\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a2c31f-6357-4b42-be58-9672ec8dcecb-tmp-dir\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-registration-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-system-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-var-lib-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72a2c31f-6357-4b42-be58-9672ec8dcecb-hosts-file\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4wd\" (UniqueName: \"kubernetes.io/projected/a53bb1dd-634e-4c2f-9e71-306c2136c837-kube-api-access-4h4wd\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-script-lib\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795009 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-system-cni-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.793975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2d8\" (UniqueName: \"kubernetes.io/projected/a40f373a-ac17-46aa-affb-41fc629af3da-kube-api-access-ft2d8\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-daemon-config\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-kubernetes\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-netns\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gjnm\" (UniqueName: \"kubernetes.io/projected/f30df3c3-736e-4aad-bd2f-44a8866b3f93-kube-api-access-4gjnm\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-cni-binary-copy\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-bin\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-kubelet-config\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysconfig\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-systemd-units\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-ovn\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.795903 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794434 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-registration-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.796652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794470 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72a2c31f-6357-4b42-be58-9672ec8dcecb-hosts-file\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.796652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794544 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-log-socket\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.796652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.794830 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-system-cni-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.796652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.795176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.796652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.795191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a2c31f-6357-4b42-be58-9672ec8dcecb-tmp-dir\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.796652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.795247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-var-lib-kubelet\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.796652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.795560 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-daemon-config\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.796652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.795606 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.797072 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-env-overrides\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.797178 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.797178 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-host\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.797178 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797135 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.797178 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a40f373a-ac17-46aa-affb-41fc629af3da-iptables-alerter-script\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.797178 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-conf-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-node-log\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-cnibin\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-modprobe-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797294 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-run\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-conf\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-socket-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-netd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnrl\" (UniqueName: \"kubernetes.io/projected/c59f3642-4311-4da7-b3e2-66c782e2b83b-kube-api-access-ggnrl\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-k8s-cni-cncf-io\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-bin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-etc-kubernetes\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.797476 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e9025a0-9daa-426a-b18d-3a04a154b693-host\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e9025a0-9daa-426a-b18d-3a04a154b693-serviceca\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797486 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-env-overrides\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e9025a0-9daa-426a-b18d-3a04a154b693-host\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797659 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-bin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-etc-kubernetes\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797790 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797809 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-cnibin\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-config\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-systemd\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797924 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-sys-fs\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjs8f\" (UniqueName: \"kubernetes.io/projected/7e9025a0-9daa-426a-b18d-3a04a154b693-kube-api-access-qjs8f\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.797975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-slash\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-dbus\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798025 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-modprobe-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:20.798118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798017 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e9025a0-9daa-426a-b18d-3a04a154b693-serviceca\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-lib-modules\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798094 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-netd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-tuned\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798125 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-netns\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40f373a-ac17-46aa-affb-41fc629af3da-host-slash\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798153 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-conf\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-device-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40f373a-ac17-46aa-affb-41fc629af3da-host-slash\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-multus-certs\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798244 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-socket-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpq4l\" (UniqueName: \"kubernetes.io/projected/67379085-0be6-451f-b409-4e47dd190877-kube-api-access-dpq4l\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-systemd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798340 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-systemd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798381 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-kubernetes\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798442 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-systemd\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.798791 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798526 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798673 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-dbus\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798734 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-ovn\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-k8s-cni-cncf-io\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-device-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-multus-certs\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798934 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-lib-modules\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.798989 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-slash\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799037 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-var-lib-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799080 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-kubelet-config\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-cni-binary-copy\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-cnibin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-multus\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799149 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-sys-fs\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-kubelet\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-host\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:20.799770 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-netns\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799324 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-bin\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-etc-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799374 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysconfig\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799395 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-systemd-units\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-system-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799425 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-multus\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-hostroot\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799468 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-cnibin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-sys\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhn2\" (UniqueName: \"kubernetes.io/projected/f11ccc81-ad15-4901-afe8-dbc49587fb90-kube-api-access-9xhn2\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799364 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799448 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-conf-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799547 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-node-log\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799585 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-kubelet\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799631 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-script-lib\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.800675 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.799671 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799686 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-etc-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.799728 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.799743 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs podName:67379085-0be6-451f-b409-4e47dd190877 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:21.299720563 +0000 UTC m=+3.086078593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs") pod "network-metrics-daemon-gwvzz" (UID: "67379085-0be6-451f-b409-4e47dd190877") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.799798 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret podName:1b37398e-d79b-4ced-a7b6-07409e2dbdcc nodeName:}" failed. No retries permitted until 2026-10-30 13:48:21.299780848 +0000 UTC m=+3.086138902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret") pod "global-pull-secret-syncer-pb5tm" (UID: "1b37398e-d79b-4ced-a7b6-07409e2dbdcc") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-agent-certs\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-konnectivity-ca\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799899 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-tmp\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799905 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-netns\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799957 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-hostroot\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.799984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-socket-dir-parent\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.800107 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-socket-dir-parent\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.800143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-kubelet\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.800223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-kubelet\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.801499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.800814 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-config\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.802052 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.801132 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a40f373a-ac17-46aa-affb-41fc629af3da-iptables-alerter-script\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.802052 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.801142 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-konnectivity-ca\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:20.802052 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.801972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.802677 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.802652 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-tuned\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.804456 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.804431 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-agent-certs\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:20.806061 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.806041 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-tmp\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.824072 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.824030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsfg\" (UniqueName: \"kubernetes.io/projected/72a2c31f-6357-4b42-be58-9672ec8dcecb-kube-api-access-dtsfg\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.825386 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.825348 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:20.825386 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.825388 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:20.825574 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.825404 2579 projected.go:196] Error preparing data for projected volume kube-api-access-p82w4 for pod openshift-network-diagnostics/network-check-target-ldf6b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:20.825574 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:20.825475 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4 podName:1f75fa3f-6f80-4ea7-a2a1-205839a11f12 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:21.325454871 +0000 UTC m=+3.111812906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p82w4" (UniqueName: "kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4") pod "network-check-target-ldf6b" (UID: "1f75fa3f-6f80-4ea7-a2a1-205839a11f12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:20.826915 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.826889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4wd\" (UniqueName: \"kubernetes.io/projected/a53bb1dd-634e-4c2f-9e71-306c2136c837-kube-api-access-4h4wd\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.830176 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.830145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpq4l\" (UniqueName: \"kubernetes.io/projected/67379085-0be6-451f-b409-4e47dd190877-kube-api-access-dpq4l\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:20.831135 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.830636 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mshk\" (UniqueName: \"kubernetes.io/projected/0f87e904-4c9c-4691-8858-dc3dfbe443af-kube-api-access-9mshk\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:20.831135 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.830903 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjs8f\" (UniqueName: \"kubernetes.io/projected/7e9025a0-9daa-426a-b18d-3a04a154b693-kube-api-access-qjs8f\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:20.831135 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.830973 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhn2\" (UniqueName: \"kubernetes.io/projected/f11ccc81-ad15-4901-afe8-dbc49587fb90-kube-api-access-9xhn2\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.831135 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.831106 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnrl\" (UniqueName: \"kubernetes.io/projected/c59f3642-4311-4da7-b3e2-66c782e2b83b-kube-api-access-ggnrl\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.831936 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.831880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gjnm\" (UniqueName: \"kubernetes.io/projected/f30df3c3-736e-4aad-bd2f-44a8866b3f93-kube-api-access-4gjnm\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.832684 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.832660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2d8\" (UniqueName: \"kubernetes.io/projected/a40f373a-ac17-46aa-affb-41fc629af3da-kube-api-access-ft2d8\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.946455 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.946410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:48:20.948600 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.948565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p7256" Oct 30 13:48:20.953206 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.953156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:20.955837 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.955780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qb4l7" Oct 30 13:48:20.958526 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:20.958490 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40f373a_ac17_46aa_affb_41fc629af3da.slice/crio-dc933f491eac7b20ed2e7a31879bdc18af87108dda08a83857cf02534067f187 WatchSource:0}: Error finding container dc933f491eac7b20ed2e7a31879bdc18af87108dda08a83857cf02534067f187: Status 404 returned error can't find the container with id dc933f491eac7b20ed2e7a31879bdc18af87108dda08a83857cf02534067f187 Oct 30 13:48:20.962359 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:20.962329 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a2c31f_6357_4b42_be58_9672ec8dcecb.slice/crio-d8e3c753d2c66acbeeb30d790d7b514dd5ddc05651808bbdfad823c0cf77f1df WatchSource:0}: Error finding container d8e3c753d2c66acbeeb30d790d7b514dd5ddc05651808bbdfad823c0cf77f1df: Status 404 returned error can't find the container with id d8e3c753d2c66acbeeb30d790d7b514dd5ddc05651808bbdfad823c0cf77f1df Oct 30 13:48:20.967219 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:20.967109 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30df3c3_736e_4aad_bd2f_44a8866b3f93.slice/crio-585e43fbe5f67219d4bb3d45bd9564d3dc8a604f4e21318542aabdbd110744b5 WatchSource:0}: Error finding container 585e43fbe5f67219d4bb3d45bd9564d3dc8a604f4e21318542aabdbd110744b5: Status 404 returned error can't find the container with id 585e43fbe5f67219d4bb3d45bd9564d3dc8a604f4e21318542aabdbd110744b5 Oct 30 13:48:20.969005 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:20.968979 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11ccc81_ad15_4901_afe8_dbc49587fb90.slice/crio-afb9afd21cc05977b91a894bf16e7f1c413489d3ec7e2a24deb753a3396e0e35 WatchSource:0}: Error finding container afb9afd21cc05977b91a894bf16e7f1c413489d3ec7e2a24deb753a3396e0e35: Status 404 returned error can't find the container with id afb9afd21cc05977b91a894bf16e7f1c413489d3ec7e2a24deb753a3396e0e35 Oct 30 13:48:20.978854 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.978821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:20.981050 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.981024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:48:20.988790 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.988762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:48:20.994185 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:20.994159 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdb85a7_7e6c_4e60_bbac_dbf902150c60.slice/crio-461354a6ffb262a94623241533d908493ffd3c85de2a91399503f640414b7a49 WatchSource:0}: Error finding container 461354a6ffb262a94623241533d908493ffd3c85de2a91399503f640414b7a49: Status 404 returned error can't find the container with id 461354a6ffb262a94623241533d908493ffd3c85de2a91399503f640414b7a49 Oct 30 13:48:20.995731 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:20.995703 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59f3642_4311_4da7_b3e2_66c782e2b83b.slice/crio-5940df8af7d84bf7c51ed70150627b5a0da2fa549fd13318dcbb2e7c63377875 WatchSource:0}: Error finding container 5940df8af7d84bf7c51ed70150627b5a0da2fa549fd13318dcbb2e7c63377875: Status 404 returned error can't find the container with id 5940df8af7d84bf7c51ed70150627b5a0da2fa549fd13318dcbb2e7c63377875 Oct 30 13:48:20.999477 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:20.999444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:48:21.003870 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.003825 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:48:21.004174 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:21.004151 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53bb1dd_634e_4c2f_9e71_306c2136c837.slice/crio-a55fa4712649caf1ac5b901febddbac3b15244d680f35749e25e8405edfbefea WatchSource:0}: Error finding container a55fa4712649caf1ac5b901febddbac3b15244d680f35749e25e8405edfbefea: Status 404 returned error can't find the container with id a55fa4712649caf1ac5b901febddbac3b15244d680f35749e25e8405edfbefea Oct 30 13:48:21.012624 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:21.012587 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f87e904_4c9c_4691_8858_dc3dfbe443af.slice/crio-307330f0d4b02f715d7a72fc494da988e592519c596639d8f82e3417ef58f436 WatchSource:0}: Error finding container 307330f0d4b02f715d7a72fc494da988e592519c596639d8f82e3417ef58f436: Status 404 returned error can't find the container with id 307330f0d4b02f715d7a72fc494da988e592519c596639d8f82e3417ef58f436 Oct 30 13:48:21.017169 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:48:21.017138 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9025a0_9daa_426a_b18d_3a04a154b693.slice/crio-9e08d19cc1d810131e0b376a84681647f90730ea81e2386a07e1e42538441af5 WatchSource:0}: Error finding container 9e08d19cc1d810131e0b376a84681647f90730ea81e2386a07e1e42538441af5: Status 404 returned error can't find the container with id 9e08d19cc1d810131e0b376a84681647f90730ea81e2386a07e1e42538441af5 Oct 30 13:48:21.304523 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.304487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:21.304718 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.304537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:21.304718 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.304692 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:21.304844 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.304754 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret podName:1b37398e-d79b-4ced-a7b6-07409e2dbdcc nodeName:}" failed. No retries permitted until 2026-10-30 13:48:22.304740001 +0000 UTC m=+4.091098044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret") pod "global-pull-secret-syncer-pb5tm" (UID: "1b37398e-d79b-4ced-a7b6-07409e2dbdcc") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:21.304844 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.304836 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:21.304952 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.304867 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs podName:67379085-0be6-451f-b409-4e47dd190877 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:22.304858688 +0000 UTC m=+4.091216731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs") pod "network-metrics-daemon-gwvzz" (UID: "67379085-0be6-451f-b409-4e47dd190877") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:21.405826 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.405787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:21.405985 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.405964 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:21.406043 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.405990 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:21.406043 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.406000 2579 projected.go:196] Error preparing data for projected volume kube-api-access-p82w4 for pod openshift-network-diagnostics/network-check-target-ldf6b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:21.406126 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.406057 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4 podName:1f75fa3f-6f80-4ea7-a2a1-205839a11f12 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:22.406043232 +0000 UTC m=+4.192401275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p82w4" (UniqueName: "kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4") pod "network-check-target-ldf6b" (UID: "1f75fa3f-6f80-4ea7-a2a1-205839a11f12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:21.657126 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.656739 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:21 +0000 UTC" deadline="2028-03-24 23:32:41.425134647 +0000 UTC" Oct 30 13:48:21.657126 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.657040 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12273h44m19.768100467s" Oct 30 13:48:21.725559 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.725526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:21.725744 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:21.725670 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:21.739984 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.739944 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" event={"ID":"0f87e904-4c9c-4691-8858-dc3dfbe443af","Type":"ContainerStarted","Data":"307330f0d4b02f715d7a72fc494da988e592519c596639d8f82e3417ef58f436"} Oct 30 13:48:21.747349 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.747310 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" event={"ID":"a53bb1dd-634e-4c2f-9e71-306c2136c837","Type":"ContainerStarted","Data":"a55fa4712649caf1ac5b901febddbac3b15244d680f35749e25e8405edfbefea"} Oct 30 13:48:21.749820 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.749789 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-stdsc" event={"ID":"5cdb85a7-7e6c-4e60-bbac-dbf902150c60","Type":"ContainerStarted","Data":"461354a6ffb262a94623241533d908493ffd3c85de2a91399503f640414b7a49"} Oct 30 13:48:21.763082 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.763043 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" event={"ID":"bda9b1f2dbe238b1af8586cc49b37f56","Type":"ContainerStarted","Data":"6d8a33d1f4c535dbcabe6eb5588b384a06ace8d04fbd03026e57e5388ce6dd98"} Oct 30 13:48:21.768244 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.768212 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pscjw" event={"ID":"7e9025a0-9daa-426a-b18d-3a04a154b693","Type":"ContainerStarted","Data":"9e08d19cc1d810131e0b376a84681647f90730ea81e2386a07e1e42538441af5"} Oct 30 13:48:21.772275 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.772189 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" event={"ID":"c59f3642-4311-4da7-b3e2-66c782e2b83b","Type":"ContainerStarted","Data":"5940df8af7d84bf7c51ed70150627b5a0da2fa549fd13318dcbb2e7c63377875"} Oct 30 13:48:21.776200 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.776145 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qb4l7" event={"ID":"f11ccc81-ad15-4901-afe8-dbc49587fb90","Type":"ContainerStarted","Data":"afb9afd21cc05977b91a894bf16e7f1c413489d3ec7e2a24deb753a3396e0e35"} Oct 30 13:48:21.788385 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.788039 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"585e43fbe5f67219d4bb3d45bd9564d3dc8a604f4e21318542aabdbd110744b5"} Oct 30 13:48:21.792208 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.792122 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p7256" event={"ID":"72a2c31f-6357-4b42-be58-9672ec8dcecb","Type":"ContainerStarted","Data":"d8e3c753d2c66acbeeb30d790d7b514dd5ddc05651808bbdfad823c0cf77f1df"} Oct 30 13:48:21.802759 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.802720 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tghzm" event={"ID":"a40f373a-ac17-46aa-affb-41fc629af3da","Type":"ContainerStarted","Data":"dc933f491eac7b20ed2e7a31879bdc18af87108dda08a83857cf02534067f187"} Oct 30 13:48:21.808774 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:21.808419 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" podStartSLOduration=1.808399675 podStartE2EDuration="1.808399675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:21.807628082 +0000 UTC m=+3.593986138" watchObservedRunningTime="2026-10-30 13:48:21.808399675 +0000 UTC m=+3.594757739" Oct 30 13:48:22.315480 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.315068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:22.315480 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.315115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:22.315480 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.315267 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:22.315480 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.315339 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret podName:1b37398e-d79b-4ced-a7b6-07409e2dbdcc nodeName:}" failed. No retries permitted until 2026-10-30 13:48:24.315319581 +0000 UTC m=+6.101677627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret") pod "global-pull-secret-syncer-pb5tm" (UID: "1b37398e-d79b-4ced-a7b6-07409e2dbdcc") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:22.315480 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.315423 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:22.315480 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.315460 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs podName:67379085-0be6-451f-b409-4e47dd190877 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:24.315447876 +0000 UTC m=+6.101805920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs") pod "network-metrics-daemon-gwvzz" (UID: "67379085-0be6-451f-b409-4e47dd190877") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:22.415590 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.415554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:22.415804 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.415786 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:22.415877 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.415813 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:22.415877 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.415827 2579 projected.go:196] Error preparing data for projected volume kube-api-access-p82w4 for pod openshift-network-diagnostics/network-check-target-ldf6b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:22.416005 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.415886 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4 podName:1f75fa3f-6f80-4ea7-a2a1-205839a11f12 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:24.415866787 +0000 UTC m=+6.202224831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p82w4" (UniqueName: "kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4") pod "network-check-target-ldf6b" (UID: "1f75fa3f-6f80-4ea7-a2a1-205839a11f12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:22.657534 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.657415 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:21 +0000 UTC" deadline="2028-07-16 07:34:04.684452472 +0000 UTC" Oct 30 13:48:22.657534 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.657484 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14993h45m42.026973199s" Oct 30 13:48:22.725001 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.724969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:22.725164 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.725120 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:22.725602 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.725581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:22.725719 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:22.725682 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:22.817909 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.817825 2579 generic.go:356] "Generic (PLEG): container finished" podID="950f8228a09238446a40b1b7667155e5" containerID="9e1fa68a73061b84bc63920775b485c771d8641b78dd81c972a9256fd6c42709" exitCode=0 Oct 30 13:48:22.818094 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:22.817978 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" event={"ID":"950f8228a09238446a40b1b7667155e5","Type":"ContainerDied","Data":"9e1fa68a73061b84bc63920775b485c771d8641b78dd81c972a9256fd6c42709"} Oct 30 13:48:23.722125 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:23.722087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:23.722639 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:23.722256 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:23.842027 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:23.841937 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" event={"ID":"950f8228a09238446a40b1b7667155e5","Type":"ContainerStarted","Data":"d0556c2bf234383a9080910ed69574b29dbd8fb1e72b9582292ac5c86212a04a"} Oct 30 13:48:23.891206 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:23.891138 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" podStartSLOduration=3.891119111 podStartE2EDuration="3.891119111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:23.889877335 +0000 UTC m=+5.676235416" watchObservedRunningTime="2026-10-30 13:48:23.891119111 +0000 UTC m=+5.677477167" Oct 30 13:48:24.337943 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:24.337905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:24.338143 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:24.337957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:24.338143 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.338136 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:24.338351 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.338203 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret podName:1b37398e-d79b-4ced-a7b6-07409e2dbdcc nodeName:}" failed. No retries permitted until 2026-10-30 13:48:28.338183497 +0000 UTC m=+10.124541541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret") pod "global-pull-secret-syncer-pb5tm" (UID: "1b37398e-d79b-4ced-a7b6-07409e2dbdcc") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:24.338650 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.338629 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:24.338739 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.338690 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs podName:67379085-0be6-451f-b409-4e47dd190877 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:28.338672529 +0000 UTC m=+10.125030573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs") pod "network-metrics-daemon-gwvzz" (UID: "67379085-0be6-451f-b409-4e47dd190877") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:24.438704 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:24.438667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:24.438918 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.438892 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:24.438918 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.438918 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:24.439051 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.438934 2579 projected.go:196] Error preparing data for projected volume kube-api-access-p82w4 for pod openshift-network-diagnostics/network-check-target-ldf6b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:24.439051 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.438999 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4 podName:1f75fa3f-6f80-4ea7-a2a1-205839a11f12 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:28.438981136 +0000 UTC m=+10.225339181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p82w4" (UniqueName: "kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4") pod "network-check-target-ldf6b" (UID: "1f75fa3f-6f80-4ea7-a2a1-205839a11f12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:24.725427 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:24.724670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:24.725427 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.724840 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:24.726254 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:24.725255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:24.726254 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:24.726207 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:24.851917 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:24.849856 2579 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="8329261c64a72f701b66f62bb24ecd32ed91c80112501becde4a5de1d2735a01" exitCode=0 Oct 30 13:48:24.851917 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:24.851490 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" event={"ID":"c59f3642-4311-4da7-b3e2-66c782e2b83b","Type":"ContainerDied","Data":"8329261c64a72f701b66f62bb24ecd32ed91c80112501becde4a5de1d2735a01"} Oct 30 13:48:25.722693 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:25.722274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:25.722693 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:25.722383 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:25.861247 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:25.861202 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-stdsc" event={"ID":"5cdb85a7-7e6c-4e60-bbac-dbf902150c60","Type":"ContainerStarted","Data":"b2ee215ad1caf8c13fd741c5d3d928805f7b68c69dca0d311588d9573d1e8a68"} Oct 30 13:48:25.909847 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:25.909242 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-stdsc" podStartSLOduration=2.496552784 podStartE2EDuration="6.909220393s" totalImagesPullingTime="4.412667609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:25.908464466 +0000 UTC m=+7.694822519" watchObservedRunningTime="2026-10-30 13:48:25.909220393 +0000 UTC m=+7.695578446" Oct 30 13:48:26.723456 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:26.723331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:26.723625 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:26.723489 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:26.723914 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:26.723740 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:26.723914 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:26.723871 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:26.870287 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:26.870212 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pscjw" event={"ID":"7e9025a0-9daa-426a-b18d-3a04a154b693","Type":"ContainerStarted","Data":"f3df779d0ee04e80d8cf4326b02af3cfa7fa1e885a7a3550b28bbe2c50ffe537"} Oct 30 13:48:26.919391 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:26.919339 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pscjw" podStartSLOduration=2.4653844400000002 podStartE2EDuration="7.919317543s" totalImagesPullingTime="5.453933103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:26.918583297 +0000 UTC m=+8.704941350" watchObservedRunningTime="2026-10-30 13:48:26.919317543 +0000 UTC m=+8.705675598" Oct 30 13:48:27.722674 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:27.722629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:27.722868 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:27.722777 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:27.874902 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:27.874721 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" event={"ID":"a53bb1dd-634e-4c2f-9e71-306c2136c837","Type":"ContainerStarted","Data":"fe2b720cdda530330e508f8d7669383edff6c0d021a60da66685cbe3c4ac33b8"} Oct 30 13:48:28.378169 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:28.377270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:28.378169 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:28.377339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:28.378169 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.377494 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:28.378169 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.377573 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret podName:1b37398e-d79b-4ced-a7b6-07409e2dbdcc nodeName:}" failed. No retries permitted until 2026-10-30 13:48:36.377549024 +0000 UTC m=+18.163907070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret") pod "global-pull-secret-syncer-pb5tm" (UID: "1b37398e-d79b-4ced-a7b6-07409e2dbdcc") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:28.378169 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.378069 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:28.378169 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.378131 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs podName:67379085-0be6-451f-b409-4e47dd190877 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:36.378115428 +0000 UTC m=+18.164473472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs") pod "network-metrics-daemon-gwvzz" (UID: "67379085-0be6-451f-b409-4e47dd190877") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:28.478577 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:28.478531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:28.478791 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.478773 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:28.478856 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.478800 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:28.478856 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.478815 2579 projected.go:196] Error preparing data for projected volume kube-api-access-p82w4 for pod openshift-network-diagnostics/network-check-target-ldf6b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:28.478929 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.478881 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4 podName:1f75fa3f-6f80-4ea7-a2a1-205839a11f12 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:36.47886056 +0000 UTC m=+18.265218611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p82w4" (UniqueName: "kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4") pod "network-check-target-ldf6b" (UID: "1f75fa3f-6f80-4ea7-a2a1-205839a11f12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:28.725492 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:28.725407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:28.725650 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.725534 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:28.725883 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:28.725852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:28.725990 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:28.725923 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:29.261752 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.261726 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:29.262632 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.262577 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:29.327799 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.327758 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 13:48:29.662297 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.661868 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:48:29.327780033Z","UUID":"a87733a4-a52a-49b8-acb3-ad2e6c9df2fc","Handler":null,"Name":"","Endpoint":""} Oct 30 13:48:29.663631 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.663611 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 13:48:29.663631 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.663636 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 13:48:29.722153 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.722122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:29.722395 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:29.722368 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:29.882306 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.882112 2579 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="05548e25457870a62d07796f9dd7063079f6a0f0216c0c09b9c37756b010c6e9" exitCode=0 Oct 30 13:48:29.882306 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.882238 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" event={"ID":"c59f3642-4311-4da7-b3e2-66c782e2b83b","Type":"ContainerDied","Data":"05548e25457870a62d07796f9dd7063079f6a0f0216c0c09b9c37756b010c6e9"} Oct 30 13:48:29.884061 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.884027 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qb4l7" event={"ID":"f11ccc81-ad15-4901-afe8-dbc49587fb90","Type":"ContainerStarted","Data":"74398972eabcac079c6b2f5e4de02aa95749cbae0fe2efa7f6b0e42956d47568"} Oct 30 13:48:29.886270 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.886245 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p7256" event={"ID":"72a2c31f-6357-4b42-be58-9672ec8dcecb","Type":"ContainerStarted","Data":"16b2f7d7d759a79226302020403f5fd2106ea202958a30c7759ac7a4a50760ac"} Oct 30 13:48:29.890849 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.890815 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" event={"ID":"a53bb1dd-634e-4c2f-9e71-306c2136c837","Type":"ContainerStarted","Data":"d7a7d78d59080e9d3eeaaa09f9b83ce537b8e0680f0ef450b7be33d2dc151daf"} Oct 30 13:48:29.983222 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:29.982454 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-p7256" podStartSLOduration=2.683413116 podStartE2EDuration="10.982436514s" totalImagesPullingTime="8.299023398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:29.981989489 +0000 UTC m=+11.768347537" watchObservedRunningTime="2026-10-30 13:48:29.982436514 +0000 UTC m=+11.768794564" Oct 30 13:48:30.028745 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:30.028685 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qb4l7" podStartSLOduration=2.815332647 podStartE2EDuration="11.028670545s" totalImagesPullingTime="8.213337898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:30.027962029 +0000 UTC m=+11.814320078" watchObservedRunningTime="2026-10-30 13:48:30.028670545 +0000 UTC m=+11.815028593" Oct 30 13:48:30.722647 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:30.722609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:30.723173 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:30.722768 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:30.723229 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:30.723216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:30.723332 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:30.723307 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:30.895703 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:30.895593 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" event={"ID":"a53bb1dd-634e-4c2f-9e71-306c2136c837","Type":"ContainerStarted","Data":"21d6a22f09aac85db6ce1d845e3b45fee9a006f0963ae38b60fbfd9f36e6960b"} Oct 30 13:48:30.898241 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:30.898211 2579 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="cf9e69647da510af7dca19cd468b24d3356c6fa931d5c923c459938edc786608" exitCode=0 Oct 30 13:48:30.898378 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:30.898305 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" event={"ID":"c59f3642-4311-4da7-b3e2-66c782e2b83b","Type":"ContainerDied","Data":"cf9e69647da510af7dca19cd468b24d3356c6fa931d5c923c459938edc786608"} Oct 30 13:48:30.944165 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:30.944105 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" podStartSLOduration=2.916216474 podStartE2EDuration="11.944085932s" totalImagesPullingTime="9.027869458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:30.943761185 +0000 UTC m=+12.730119234" watchObservedRunningTime="2026-10-30 13:48:30.944085932 +0000 UTC m=+12.730443983" Oct 30 13:48:31.722383 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:31.722344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:31.722559 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:31.722526 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:31.901813 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:31.901777 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tghzm" event={"ID":"a40f373a-ac17-46aa-affb-41fc629af3da","Type":"ContainerStarted","Data":"9b060800a62a9a40d26acf8f93fd2f1f292dea52ed2799d27f5b9a37bd71131d"} Oct 30 13:48:31.903642 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:31.903615 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" event={"ID":"0f87e904-4c9c-4691-8858-dc3dfbe443af","Type":"ContainerStarted","Data":"a7a27c071aded3cdebb342b6a987bab848b4131dc967b2cab6d41c9a79b6c08c"} Oct 30 13:48:31.947710 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:31.947647 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tghzm" podStartSLOduration=4.351327856 podStartE2EDuration="12.947629291s" totalImagesPullingTime="8.596301435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:31.947177258 +0000 UTC m=+13.733535306" watchObservedRunningTime="2026-10-30 13:48:31.947629291 +0000 UTC m=+13.733987342" Oct 30 13:48:31.995701 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:31.995586 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" podStartSLOduration=2.280625642 podStartE2EDuration="12.995567261s" totalImagesPullingTime="10.714941619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:31.995217612 +0000 UTC m=+13.781575662" watchObservedRunningTime="2026-10-30 13:48:31.995567261 +0000 UTC m=+13.781925312" Oct 30 13:48:32.724688 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:32.724656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:32.724851 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:32.724777 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:32.725173 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:32.725153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:32.725276 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:32.725263 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:32.908068 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:32.908039 2579 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="b06d25e3dd0c44e8ac4d999437a27c2ea33cba0b4ccdb8ed56584de637cefc5c" exitCode=0 Oct 30 13:48:32.908558 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:32.908106 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" event={"ID":"c59f3642-4311-4da7-b3e2-66c782e2b83b","Type":"ContainerDied","Data":"b06d25e3dd0c44e8ac4d999437a27c2ea33cba0b4ccdb8ed56584de637cefc5c"} Oct 30 13:48:33.722415 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:33.722147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:33.722568 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:33.722524 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:34.722583 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:34.722541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:34.722583 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:34.722569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:34.723013 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:34.722670 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:34.723013 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:34.722781 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:34.915995 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:34.915907 2579 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="e60d1665e8d9e5b3d5c9139ecfca33a123384b88cb5f0690ca828cf9529090a7" exitCode=0 Oct 30 13:48:34.915995 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:34.915971 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" event={"ID":"c59f3642-4311-4da7-b3e2-66c782e2b83b","Type":"ContainerDied","Data":"e60d1665e8d9e5b3d5c9139ecfca33a123384b88cb5f0690ca828cf9529090a7"} Oct 30 13:48:35.722638 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:35.722604 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:35.723026 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:35.722711 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:35.920725 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:35.920688 2579 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="4bca3408976837265c19d7673a09253998a6c0992873052d4445ad1489357b2c" exitCode=0 Oct 30 13:48:35.920890 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:35.920741 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" event={"ID":"c59f3642-4311-4da7-b3e2-66c782e2b83b","Type":"ContainerDied","Data":"4bca3408976837265c19d7673a09253998a6c0992873052d4445ad1489357b2c"} Oct 30 13:48:36.442378 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.442217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:36.442472 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.442391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:36.442472 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.442353 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:36.442532 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.442483 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:36.442532 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.442492 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs podName:67379085-0be6-451f-b409-4e47dd190877 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.442478929 +0000 UTC m=+34.228836957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs") pod "network-metrics-daemon-gwvzz" (UID: "67379085-0be6-451f-b409-4e47dd190877") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:36.442532 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.442521 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret podName:1b37398e-d79b-4ced-a7b6-07409e2dbdcc nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.442510542 +0000 UTC m=+34.228868604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret") pod "global-pull-secret-syncer-pb5tm" (UID: "1b37398e-d79b-4ced-a7b6-07409e2dbdcc") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:36.543171 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.543145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:36.543349 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.543309 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:36.543349 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.543332 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:36.543349 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.543346 2579 projected.go:196] Error preparing data for projected volume kube-api-access-p82w4 for pod openshift-network-diagnostics/network-check-target-ldf6b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:36.543453 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.543391 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4 podName:1f75fa3f-6f80-4ea7-a2a1-205839a11f12 nodeName:}" failed. No retries permitted until 2026-10-30 13:48:52.543377895 +0000 UTC m=+34.329735922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p82w4" (UniqueName: "kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4") pod "network-check-target-ldf6b" (UID: "1f75fa3f-6f80-4ea7-a2a1-205839a11f12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:36.722424 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.722385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:36.722599 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.722505 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:36.722599 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.722556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:36.722670 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:36.722650 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:36.926386 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.926294 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" event={"ID":"c59f3642-4311-4da7-b3e2-66c782e2b83b","Type":"ContainerStarted","Data":"dfae11e0ec61c8fae41954a7834e9319991eb186a5ad10e944bca5b52e21a9fa"} Oct 30 13:48:36.929162 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.929136 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"7c8337c299d408efcb36efc4cb018928733478a9311d1ae476cbb829316b5af2"} Oct 30 13:48:36.929275 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.929209 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"f72cf3960805fb779671ecc5012c5dc05eb68c8a79ce79ebd92e023bc85a2e2e"} Oct 30 13:48:36.929275 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.929240 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"92ead115c3de24b4f332002226441c88abbbbfcfb083a5c877722929792ad4c1"} Oct 30 13:48:36.929275 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.929268 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"15725c9b8adb8ea3be9644c7f2654ab4c4521457d92a68c935601cde814c667e"} Oct 30 13:48:36.929375 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.929295 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"8be66752b96d0f07b734f96e48ae1340c9c44c777f89cf8182f3716fb960d934"} Oct 30 13:48:36.929375 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.929320 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"f1d8bf8ea877674c4b8b7d77c6be5c795c1791e0618572b95824edbce32bb292"} Oct 30 13:48:36.979330 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:36.979280 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6wf4d" podStartSLOduration=6.926231532 podStartE2EDuration="17.979264907s" totalImagesPullingTime="11.053033375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:36.97873137 +0000 UTC m=+18.765089418" watchObservedRunningTime="2026-10-30 13:48:36.979264907 +0000 UTC m=+18.765622959" Oct 30 13:48:37.721798 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:37.721747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:37.721961 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:37.721860 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:38.723184 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:38.723148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:38.723662 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:38.723276 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:38.723662 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:38.723327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:38.723662 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:38.723443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:38.937676 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:38.937636 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"c87c4987d3203a1bdef217c9a59103221353176d1cd8d3d2ec0cbd3d1986fcee"} Oct 30 13:48:39.722314 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:39.722282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:39.722484 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:39.722374 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:40.722855 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:40.722834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:40.722855 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:40.722847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:40.723382 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:40.722973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:40.723382 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:40.723027 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:40.945757 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:40.945723 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" event={"ID":"f30df3c3-736e-4aad-bd2f-44a8866b3f93","Type":"ContainerStarted","Data":"c822de1d7628badc708ce0b086718f05b23c481e2b408b80d307fb2e5c0edb40"} Oct 30 13:48:40.946010 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:40.945980 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:40.946140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:40.946050 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:40.962170 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:40.962144 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:41.043676 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:41.043626 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" podStartSLOduration=6.781202588 podStartE2EDuration="22.043612383s" totalImagesPullingTime="15.262409795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:41.042894166 +0000 UTC m=+22.829252228" watchObservedRunningTime="2026-10-30 13:48:41.043612383 +0000 UTC m=+22.829970429" Oct 30 13:48:41.721858 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:41.721823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:41.722054 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:41.721954 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:41.949619 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:41.949586 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:41.965865 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:41.965838 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:48:42.720694 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:42.720617 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ldf6b"] Oct 30 13:48:42.720857 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:42.720769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:42.720895 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:42.720862 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:42.723851 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:42.723827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:42.723955 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:42.723917 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:42.726283 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:42.726265 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pb5tm"] Oct 30 13:48:42.726384 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:42.726301 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gwvzz"] Oct 30 13:48:42.726384 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:42.726357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:42.726442 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:42.726423 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:42.951487 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:42.951458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:42.951943 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:42.951574 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:44.721809 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:44.721771 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:44.722273 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:44.721771 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:44.722273 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:44.721934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:44.722273 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:44.721771 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:44.722273 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:44.721993 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:44.722273 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:44.722043 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:46.722463 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:46.722434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:46.722899 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:46.722471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:46.722899 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:46.722555 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:46.722899 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:46.722582 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:46.722899 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:46.722630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:46.722899 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:46.722683 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:48.723800 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:48.723547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:48.724226 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:48.723606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:48.724226 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:48.723904 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:48.724226 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:48.723622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:48.724226 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:48.724004 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:48.724226 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:48.724064 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:50.665989 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:50.665951 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:50.666436 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:50.666128 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:48:50.666584 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:50.666570 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:48:50.721910 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:50.721882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:50.722050 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:50.721921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:50.722050 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:50.721998 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:50.722050 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:50.722028 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:50.722155 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:50.722071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:50.722155 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:50.722127 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:51.909524 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:51.909495 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p7256_72a2c31f-6357-4b42-be58-9672ec8dcecb/dns-node-resolver/0.log" Oct 30 13:48:52.453327 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:52.453292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:52.453327 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:52.453326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:52.453533 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.453431 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:52.453533 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.453432 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:52.453533 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.453483 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs podName:67379085-0be6-451f-b409-4e47dd190877 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:24.453469332 +0000 UTC m=+66.239827360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs") pod "network-metrics-daemon-gwvzz" (UID: "67379085-0be6-451f-b409-4e47dd190877") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 13:48:52.453533 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.453497 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret podName:1b37398e-d79b-4ced-a7b6-07409e2dbdcc nodeName:}" failed. No retries permitted until 2026-10-30 13:49:24.4534908 +0000 UTC m=+66.239848827 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret") pod "global-pull-secret-syncer-pb5tm" (UID: "1b37398e-d79b-4ced-a7b6-07409e2dbdcc") : object "kube-system"/"original-pull-secret" not registered Oct 30 13:48:52.553940 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:52.553903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:52.554092 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.554048 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 13:48:52.554092 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.554069 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 13:48:52.554092 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.554078 2579 projected.go:196] Error preparing data for projected volume kube-api-access-p82w4 for pod openshift-network-diagnostics/network-check-target-ldf6b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:52.554230 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.554125 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4 podName:1f75fa3f-6f80-4ea7-a2a1-205839a11f12 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:24.554111926 +0000 UTC m=+66.340469967 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p82w4" (UniqueName: "kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4") pod "network-check-target-ldf6b" (UID: "1f75fa3f-6f80-4ea7-a2a1-205839a11f12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 13:48:52.721995 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:52.721913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:52.722162 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.722027 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:52.722162 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:52.722098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:52.722307 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.722185 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:52.722307 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:52.722266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:52.722413 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:52.722349 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:52.841786 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:52.841760 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pscjw_7e9025a0-9daa-426a-b18d-3a04a154b693/node-ca/0.log" Oct 30 13:48:54.722482 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:54.722442 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:54.722482 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:54.722472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:54.722886 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:54.722508 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:54.722886 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:54.722602 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:54.722886 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:54.722720 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:54.722886 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:54.722789 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:56.721872 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:56.721843 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:56.722261 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:56.721949 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:56.722261 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:56.722026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:56.722261 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:56.722108 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:48:56.722261 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:56.722160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:56.722261 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:56.722232 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:58.723540 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:58.723510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:48:58.723957 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:58.723636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:48:58.723957 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:58.723657 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:48:58.723957 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:48:58.723730 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:48:58.723957 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:58.723735 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:48:58.723957 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:48:58.723812 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:49:00.722522 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:00.722486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:00.722963 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:00.722485 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:00.722963 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:00.722595 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:49:00.722963 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:00.722485 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:00.722963 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:00.722675 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:49:00.722963 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:00.722782 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:49:02.722619 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:02.722576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:02.723135 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:02.722596 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:02.723135 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:02.722701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:02.723135 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:02.722699 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:49:02.723135 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:02.722786 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:49:02.723135 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:02.722874 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:49:04.722184 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:04.722149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:04.722597 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:04.722163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:04.722597 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:04.722274 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:49:04.722597 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:04.722164 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:04.722597 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:04.722325 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:49:04.722597 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:04.722388 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:49:06.721953 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:06.721927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:06.722307 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:06.721931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:06.722307 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:06.722030 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf6b" podUID="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" Oct 30 13:49:06.722307 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:06.721936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:06.722307 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:06.722106 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pb5tm" podUID="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" Oct 30 13:49:06.722307 ip-10-0-131-200 kubenswrapper[2579]: E1030 13:49:06.722207 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwvzz" podUID="67379085-0be6-451f-b409-4e47dd190877" Oct 30 13:49:07.016647 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.016618 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeReady" Oct 30 13:49:07.016799 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.016732 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 13:49:07.084118 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.084084 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7n7tm"] Oct 30 13:49:07.085925 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.085905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.091488 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.091468 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m5qfs"] Oct 30 13:49:07.093101 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.093081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:07.106873 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.106853 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7n7tm"] Oct 30 13:49:07.110229 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.110207 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 13:49:07.110422 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.110405 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7cls\"" Oct 30 13:49:07.110497 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.110472 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 13:49:07.110842 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.110821 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6ltg7"] Oct 30 13:49:07.112235 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.112217 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 13:49:07.112746 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.112728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.113321 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.113303 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m5qfs"] Oct 30 13:49:07.138361 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.138342 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6ltg7"] Oct 30 13:49:07.167018 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvz6\" (UniqueName: \"kubernetes.io/projected/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-kube-api-access-dsvz6\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.167131 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c082c71c-c2af-4b9d-bde3-35e11d42fb51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.167131 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qvh\" (UniqueName: \"kubernetes.io/projected/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-api-access-s5qvh\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.167131 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082c71c-c2af-4b9d-bde3-35e11d42fb51-data-volume\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.167131 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-metrics-tls\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.167322 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmqsp\" (UniqueName: \"kubernetes.io/projected/e9f9b8de-895f-4e27-b089-0f97cfefdff8-kube-api-access-cmqsp\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:07.167322 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.167483 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f9b8de-895f-4e27-b089-0f97cfefdff8-cert\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:07.167483 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c082c71c-c2af-4b9d-bde3-35e11d42fb51-crio-socket\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.167483 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-config-volume\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.167483 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.167437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-tmp-dir\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.212338 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.212307 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 13:49:07.212338 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.212323 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 13:49:07.212488 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.212307 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 13:49:07.212488 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.212431 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w6ztc\"" Oct 30 13:49:07.212488 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.212458 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 13:49:07.212488 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.212474 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 13:49:07.212488 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.212458 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-snt8q\"" Oct 30 13:49:07.212641 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.212482 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 13:49:07.268259 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c082c71c-c2af-4b9d-bde3-35e11d42fb51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.268259 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qvh\" (UniqueName: \"kubernetes.io/projected/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-api-access-s5qvh\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.268259 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082c71c-c2af-4b9d-bde3-35e11d42fb51-data-volume\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.268453 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-metrics-tls\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.268453 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmqsp\" (UniqueName: \"kubernetes.io/projected/e9f9b8de-895f-4e27-b089-0f97cfefdff8-kube-api-access-cmqsp\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:07.268453 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.268453 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f9b8de-895f-4e27-b089-0f97cfefdff8-cert\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:07.268639 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c082c71c-c2af-4b9d-bde3-35e11d42fb51-crio-socket\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.268639 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-config-volume\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.268639 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-tmp-dir\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.268639 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvz6\" (UniqueName: \"kubernetes.io/projected/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-kube-api-access-dsvz6\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.268827 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268711 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082c71c-c2af-4b9d-bde3-35e11d42fb51-data-volume\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.268877 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c082c71c-c2af-4b9d-bde3-35e11d42fb51-crio-socket\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.268966 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.268945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.269170 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.269151 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-tmp-dir\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.269377 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.269350 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-config-volume\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.272076 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.272057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-metrics-tls\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.272165 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.272104 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c082c71c-c2af-4b9d-bde3-35e11d42fb51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.272242 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.272167 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f9b8de-895f-4e27-b089-0f97cfefdff8-cert\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:07.294243 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.294219 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qvh\" (UniqueName: \"kubernetes.io/projected/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-api-access-s5qvh\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.295066 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.295037 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvz6\" (UniqueName: \"kubernetes.io/projected/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-kube-api-access-dsvz6\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.295156 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.295141 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmqsp\" (UniqueName: \"kubernetes.io/projected/e9f9b8de-895f-4e27-b089-0f97cfefdff8-kube-api-access-cmqsp\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:07.396934 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.396896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:07.402520 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.402502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:07.424331 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.424306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:07.719937 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.719909 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m5qfs"] Oct 30 13:49:07.721000 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:07.720977 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f9b8de_895f_4e27_b089_0f97cfefdff8.slice/crio-4ff2a8b95f6634d328e971d1c6be6484e6c41c18d3e2fc269be006d38c54a004 WatchSource:0}: Error finding container 4ff2a8b95f6634d328e971d1c6be6484e6c41c18d3e2fc269be006d38c54a004: Status 404 returned error can't find the container with id 4ff2a8b95f6634d328e971d1c6be6484e6c41c18d3e2fc269be006d38c54a004 Oct 30 13:49:07.729946 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.729927 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7n7tm"] Oct 30 13:49:07.731841 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:07.731818 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80b9b41_b573_4246_b0ca_fd5e7297e6f0.slice/crio-f6ddd1f9a5297f1b2b31cabbe78e24dcc066e3311f6b515783eea7abcd28170b WatchSource:0}: Error finding container f6ddd1f9a5297f1b2b31cabbe78e24dcc066e3311f6b515783eea7abcd28170b: Status 404 returned error can't find the container with id f6ddd1f9a5297f1b2b31cabbe78e24dcc066e3311f6b515783eea7abcd28170b Oct 30 13:49:07.750786 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:07.750767 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6ltg7"] Oct 30 13:49:07.752259 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:07.752240 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc082c71c_c2af_4b9d_bde3_35e11d42fb51.slice/crio-9a187132a53451c593de5a474312eb47b6f79e64d2d0fd04e87c0c58567261b9 WatchSource:0}: Error finding container 9a187132a53451c593de5a474312eb47b6f79e64d2d0fd04e87c0c58567261b9: Status 404 returned error can't find the container with id 9a187132a53451c593de5a474312eb47b6f79e64d2d0fd04e87c0c58567261b9 Oct 30 13:49:08.004039 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.003953 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ltg7" event={"ID":"c082c71c-c2af-4b9d-bde3-35e11d42fb51","Type":"ContainerStarted","Data":"bb613d78dcafd047f0fa35407d2e5800265203f390677b1a1b9979c5e0fb63f1"} Oct 30 13:49:08.004206 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.004042 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ltg7" event={"ID":"c082c71c-c2af-4b9d-bde3-35e11d42fb51","Type":"ContainerStarted","Data":"9a187132a53451c593de5a474312eb47b6f79e64d2d0fd04e87c0c58567261b9"} Oct 30 13:49:08.005108 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.005078 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m5qfs" event={"ID":"e9f9b8de-895f-4e27-b089-0f97cfefdff8","Type":"ContainerStarted","Data":"4ff2a8b95f6634d328e971d1c6be6484e6c41c18d3e2fc269be006d38c54a004"} Oct 30 13:49:08.006171 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.006149 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7n7tm" event={"ID":"c80b9b41-b573-4246-b0ca-fd5e7297e6f0","Type":"ContainerStarted","Data":"f6ddd1f9a5297f1b2b31cabbe78e24dcc066e3311f6b515783eea7abcd28170b"} Oct 30 13:49:08.662211 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.662108 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bcd85777b-rpw8q"] Oct 30 13:49:08.664313 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.664290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.680445 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.680418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.680566 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.680462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.680566 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.680494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.680675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.680651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.680727 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.680683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.680727 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.680697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.680727 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.680718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.683628 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.683606 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 13:49:08.683784 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.683768 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 13:49:08.687212 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.685803 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 13:49:08.687212 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.685943 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 13:49:08.687212 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.686106 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nh789\"" Oct 30 13:49:08.698212 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.698173 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcd85777b-rpw8q"] Oct 30 13:49:08.703854 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.703834 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 13:49:08.704056 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.704041 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 13:49:08.721864 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.721843 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:08.721864 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.721858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:08.722465 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.722203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:08.781968 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.781939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.781968 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.781967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.782370 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.781982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.782370 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.782008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.782370 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.782026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.782370 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.782046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.782370 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.782095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.782623 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.782586 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.783036 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.783019 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.783289 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.783261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.783975 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.783865 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.784749 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.784302 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 13:49:08.784749 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.784431 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 13:49:08.784749 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.784596 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 13:49:08.784749 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.784613 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 13:49:08.784749 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.784613 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w8shz\"" Oct 30 13:49:08.784749 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.784618 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gw59s\"" Oct 30 13:49:08.784749 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.784676 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smj8j\"" Oct 30 13:49:08.784749 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.784687 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 13:49:08.786648 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.786626 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.797596 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.797571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.811803 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.811770 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:08.981087 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:08.980822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:09.016745 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.016647 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ltg7" event={"ID":"c082c71c-c2af-4b9d-bde3-35e11d42fb51","Type":"ContainerStarted","Data":"c76b52023f429cf42bf525ca0577dc2b922bddddbd8c818a64fc2687e7f95a99"} Oct 30 13:49:09.293347 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.293225 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcd85777b-rpw8q"] Oct 30 13:49:09.301400 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:09.296466 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde19c6ff_20d2_4b1b_a73d_418a7b4fabf6.slice/crio-3df94a1d75c19881b6612fb5902ea860d3574702a8f4d289f07f476189b2c655 WatchSource:0}: Error finding container 3df94a1d75c19881b6612fb5902ea860d3574702a8f4d289f07f476189b2c655: Status 404 returned error can't find the container with id 3df94a1d75c19881b6612fb5902ea860d3574702a8f4d289f07f476189b2c655 Oct 30 13:49:09.409800 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.408827 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4dblz"] Oct 30 13:49:09.411130 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.411099 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.431865 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.431372 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 13:49:09.431865 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.431451 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 13:49:09.431865 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.431521 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4dblz"] Oct 30 13:49:09.431865 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.431661 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 13:49:09.431865 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.431738 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lkhsk\"" Oct 30 13:49:09.431865 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.431824 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 13:49:09.432228 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.432012 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 13:49:09.486839 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.486800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45341aa5-a0c9-4241-ae17-fa2f1b223f74-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.487001 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.486857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.487001 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.486963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtp2p\" (UniqueName: \"kubernetes.io/projected/45341aa5-a0c9-4241-ae17-fa2f1b223f74-kube-api-access-xtp2p\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.487138 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.487115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.588339 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.588310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45341aa5-a0c9-4241-ae17-fa2f1b223f74-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.588439 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.588357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.588439 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.588395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtp2p\" (UniqueName: \"kubernetes.io/projected/45341aa5-a0c9-4241-ae17-fa2f1b223f74-kube-api-access-xtp2p\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.588533 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.588481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.589116 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.589089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45341aa5-a0c9-4241-ae17-fa2f1b223f74-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.591699 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.591651 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.591803 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.591779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.615320 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.615244 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtp2p\" (UniqueName: \"kubernetes.io/projected/45341aa5-a0c9-4241-ae17-fa2f1b223f74-kube-api-access-xtp2p\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:09.725483 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:09.725456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:10.021302 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:10.021145 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7n7tm" event={"ID":"c80b9b41-b573-4246-b0ca-fd5e7297e6f0","Type":"ContainerStarted","Data":"1cf826ff9efb268454025fcb3015f43747c6809a66fd9ef3addd59779be86bd5"} Oct 30 13:49:10.021302 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:10.021256 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7n7tm" event={"ID":"c80b9b41-b573-4246-b0ca-fd5e7297e6f0","Type":"ContainerStarted","Data":"8c972147d0016fbceb63406fc8a33a2274e25abfbf633c8efb1f34fda76b45e0"} Oct 30 13:49:10.021302 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:10.021280 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:10.023121 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:10.023080 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd85777b-rpw8q" event={"ID":"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6","Type":"ContainerStarted","Data":"3df94a1d75c19881b6612fb5902ea860d3574702a8f4d289f07f476189b2c655"} Oct 30 13:49:10.036428 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:10.036409 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-4dblz"] Oct 30 13:49:10.045245 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:10.045217 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45341aa5_a0c9_4241_ae17_fa2f1b223f74.slice/crio-5121026b463798d4c27877cdf06084756d832e3ac9e6032893c057e9b93652e3 WatchSource:0}: Error finding container 5121026b463798d4c27877cdf06084756d832e3ac9e6032893c057e9b93652e3: Status 404 returned error can't find the container with id 5121026b463798d4c27877cdf06084756d832e3ac9e6032893c057e9b93652e3 Oct 30 13:49:10.067968 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:10.067894 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-7n7tm" podStartSLOduration=1.234493502 podStartE2EDuration="3.067877316s" totalImagesPullingTime="1.833383814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:10.066984876 +0000 UTC m=+51.853342937" watchObservedRunningTime="2026-10-30 13:49:10.067877316 +0000 UTC m=+51.854235366" Oct 30 13:49:11.027953 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:11.027916 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" event={"ID":"45341aa5-a0c9-4241-ae17-fa2f1b223f74","Type":"ContainerStarted","Data":"5121026b463798d4c27877cdf06084756d832e3ac9e6032893c057e9b93652e3"} Oct 30 13:49:11.031437 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:11.030859 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ltg7" event={"ID":"c082c71c-c2af-4b9d-bde3-35e11d42fb51","Type":"ContainerStarted","Data":"802e154ef1052093185809ea75e39f658532ed119e9c5e36f44aee6ee8212e7e"} Oct 30 13:49:11.035600 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:11.035547 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m5qfs" event={"ID":"e9f9b8de-895f-4e27-b089-0f97cfefdff8","Type":"ContainerStarted","Data":"f394f405e5a581075165bb864bf92775166f63e16dc2d55d69c52c9c2d894ece"} Oct 30 13:49:11.097699 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:11.097645 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6ltg7" podStartSLOduration=1.983304601 podStartE2EDuration="4.097622525s" totalImagesPullingTime="2.114317924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:49:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:11.083588705 +0000 UTC m=+52.869946774" watchObservedRunningTime="2026-10-30 13:49:11.097622525 +0000 UTC m=+52.883980574" Oct 30 13:49:11.134164 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:11.134099 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m5qfs" podStartSLOduration=0.912717814 podStartE2EDuration="4.134081919s" totalImagesPullingTime="3.221364105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:11.133300685 +0000 UTC m=+52.919658735" watchObservedRunningTime="2026-10-30 13:49:11.134081919 +0000 UTC m=+52.920439969" Oct 30 13:49:12.040106 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:12.040052 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" event={"ID":"45341aa5-a0c9-4241-ae17-fa2f1b223f74","Type":"ContainerStarted","Data":"bc6586808529c5acf06137984266663f9b45a8ae525427a341cc1153dfb60827"} Oct 30 13:49:12.040612 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:12.040126 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" event={"ID":"45341aa5-a0c9-4241-ae17-fa2f1b223f74","Type":"ContainerStarted","Data":"961c10f36c6c13b7954d9fbf28a727b5ee076a47b33058ba731ca8ac96d1ffaf"} Oct 30 13:49:12.086509 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:12.086454 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" podStartSLOduration=1.8025145230000001 podStartE2EDuration="3.086441104s" totalImagesPullingTime="1.283926581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:12.085709036 +0000 UTC m=+53.872067084" watchObservedRunningTime="2026-10-30 13:49:12.086441104 +0000 UTC m=+53.872799153" Oct 30 13:49:13.047150 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:13.047107 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd85777b-rpw8q" event={"ID":"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6","Type":"ContainerStarted","Data":"c2a51c895230dde8d9b5e0a162ff9eec9c7d7e7ae4978d8370137369dd0f949b"} Oct 30 13:49:13.098059 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:13.098008 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7bcd85777b-rpw8q" podStartSLOduration=2.17735833 podStartE2EDuration="5.097994917s" totalImagesPullingTime="2.920636587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:13.097458119 +0000 UTC m=+54.883816167" watchObservedRunningTime="2026-10-30 13:49:13.097994917 +0000 UTC m=+54.884352965" Oct 30 13:49:13.964121 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:13.964089 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:14.011459 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.011433 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj"] Oct 30 13:49:14.013565 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.013548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.025669 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.025647 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dsrlt"] Oct 30 13:49:14.027721 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.027707 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.038717 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.038688 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj"] Oct 30 13:49:14.103430 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.103400 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 13:49:14.103803 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.103568 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 13:49:14.103997 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.103979 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hdbtp\"" Oct 30 13:49:14.103997 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.103991 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 13:49:14.104152 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.104011 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 13:49:14.104152 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.104022 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8r92s\"" Oct 30 13:49:14.115113 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.115096 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 13:49:14.115251 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.115237 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 13:49:14.125630 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.125609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-wtmp\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.125718 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.125641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-tls\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.125718 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.125668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.125718 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.125686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-metrics-client-ca\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.125825 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.125811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.125860 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.125847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.125930 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.125908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.126014 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.125997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwsv\" (UniqueName: \"kubernetes.io/projected/d413c8fe-be32-46db-9326-5cfb42fc987b-kube-api-access-gcwsv\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.126059 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.126044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.126135 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.126116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.126215 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.126146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-textfile\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.126215 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.126185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-sys\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.126331 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.126257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7ft\" (UniqueName: \"kubernetes.io/projected/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-api-access-qw7ft\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.126331 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.126285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.126432 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.126333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-root\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.227612 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.227612 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-root\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.227612 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-wtmp\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.227853 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-tls\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.227853 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227658 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-root\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.227853 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.227853 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227756 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-wtmp\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.227853 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-metrics-client-ca\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.227853 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwsv\" (UniqueName: \"kubernetes.io/projected/d413c8fe-be32-46db-9326-5cfb42fc987b-kube-api-access-gcwsv\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.227999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-textfile\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-sys\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7ft\" (UniqueName: \"kubernetes.io/projected/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-api-access-qw7ft\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.228140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228118 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.228591 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228451 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-metrics-client-ca\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.228591 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228533 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.228701 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228662 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.228754 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228728 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-sys\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.228809 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.228763 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-textfile\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.229077 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.229046 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.230091 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.230072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.230394 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.230373 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-tls\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.230616 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.230596 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.230662 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.230640 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.254751 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.254722 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwsv\" (UniqueName: \"kubernetes.io/projected/d413c8fe-be32-46db-9326-5cfb42fc987b-kube-api-access-gcwsv\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.255128 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.255111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7ft\" (UniqueName: \"kubernetes.io/projected/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-api-access-qw7ft\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.325748 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.325712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:14.339376 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.339346 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:14.347665 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:14.347627 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd413c8fe_be32_46db_9326_5cfb42fc987b.slice/crio-3c116049c5f0051c4c255c94f95aebaa27a033149b9dbe649dcd99b2b3370f01 WatchSource:0}: Error finding container 3c116049c5f0051c4c255c94f95aebaa27a033149b9dbe649dcd99b2b3370f01: Status 404 returned error can't find the container with id 3c116049c5f0051c4c255c94f95aebaa27a033149b9dbe649dcd99b2b3370f01 Oct 30 13:49:14.633916 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:14.633888 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj"] Oct 30 13:49:14.635945 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:14.635917 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80844f10_0b8f_4f5e_bbba_4a93bb0a8626.slice/crio-9e0458ec69773388dca1845aea143308502241a12b9c8d0981f677f9e946638b WatchSource:0}: Error finding container 9e0458ec69773388dca1845aea143308502241a12b9c8d0981f677f9e946638b: Status 404 returned error can't find the container with id 9e0458ec69773388dca1845aea143308502241a12b9c8d0981f677f9e946638b Oct 30 13:49:15.056652 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.056609 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsrlt" event={"ID":"d413c8fe-be32-46db-9326-5cfb42fc987b","Type":"ContainerStarted","Data":"3c116049c5f0051c4c255c94f95aebaa27a033149b9dbe649dcd99b2b3370f01"} Oct 30 13:49:15.058152 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.058123 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" event={"ID":"80844f10-0b8f-4f5e-bbba-4a93bb0a8626","Type":"ContainerStarted","Data":"9e0458ec69773388dca1845aea143308502241a12b9c8d0981f677f9e946638b"} Oct 30 13:49:15.468075 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.468041 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:15.471047 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.471026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.489615 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.489477 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 13:49:15.489615 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.489477 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 13:49:15.489615 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.489509 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 13:49:15.489615 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.489487 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 13:49:15.489615 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.489483 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6lwlb\"" Oct 30 13:49:15.489615 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.489556 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 13:49:15.489615 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.489593 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 13:49:15.503363 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.503330 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:15.516265 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.516239 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 13:49:15.538819 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.538797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.538938 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.538827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.538938 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.538845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szdqr\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.538938 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.538869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539106 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539106 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539222 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539222 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539322 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539322 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539322 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539322 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.539510 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.539362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.593861 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.593817 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 13:49:15.593861 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.593821 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 13:49:15.640529 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640529 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640706 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdqr\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640706 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640706 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640706 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640706 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640914 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640914 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640914 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640914 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640914 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.640914 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.641131 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.640954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.641499 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.641216 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.642572 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.642520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.643833 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.643798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.644232 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.644209 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.644349 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.644301 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.644659 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.644623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.644659 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.644644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.645756 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.645706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.646076 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.646049 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.646416 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.646390 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.646557 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.646535 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.682600 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.682579 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdqr\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:15.791496 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:15.791468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:16.063471 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.063436 2579 generic.go:356] "Generic (PLEG): container finished" podID="d413c8fe-be32-46db-9326-5cfb42fc987b" containerID="383796b376655add5e6ec7133964db66b5c2a332319d03cf43c40daf215466d6" exitCode=0 Oct 30 13:49:16.063614 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.063514 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsrlt" event={"ID":"d413c8fe-be32-46db-9326-5cfb42fc987b","Type":"ContainerDied","Data":"383796b376655add5e6ec7133964db66b5c2a332319d03cf43c40daf215466d6"} Oct 30 13:49:16.065799 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.065768 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" event={"ID":"80844f10-0b8f-4f5e-bbba-4a93bb0a8626","Type":"ContainerStarted","Data":"68549cb6e651abe444f60b7a7d3a487572f2eeb4d203148ad80d9e68484622fb"} Oct 30 13:49:16.065894 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.065830 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" event={"ID":"80844f10-0b8f-4f5e-bbba-4a93bb0a8626","Type":"ContainerStarted","Data":"e5e10589fe826bdefb7b08b7bb2e5c1212f148883930ca792e93288fca33710a"} Oct 30 13:49:16.065894 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.065866 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" event={"ID":"80844f10-0b8f-4f5e-bbba-4a93bb0a8626","Type":"ContainerStarted","Data":"65ecb1eaf9070a788a133a5603566682ae7ed4ac678d921dd7dd1cc0c57cd4a6"} Oct 30 13:49:16.125401 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.125377 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:16.140239 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:16.140213 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc254d0c_5015_4683_98fa_c423c989b20c.slice/crio-c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35 WatchSource:0}: Error finding container c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35: Status 404 returned error can't find the container with id c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35 Oct 30 13:49:16.159016 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.158970 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" podStartSLOduration=2.071302981 podStartE2EDuration="3.158957312s" totalImagesPullingTime="1.087654331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:16.158601822 +0000 UTC m=+57.944959882" watchObservedRunningTime="2026-10-30 13:49:16.158957312 +0000 UTC m=+57.945315352" Oct 30 13:49:16.250624 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.250586 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79"] Oct 30 13:49:16.258796 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.258764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.277754 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.277730 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qrpwr\"" Oct 30 13:49:16.277754 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.277739 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 13:49:16.277943 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.277778 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 13:49:16.277943 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.277818 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 13:49:16.277943 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.277834 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 13:49:16.278090 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.278047 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 13:49:16.278090 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.278056 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-53r2k88csueue\"" Oct 30 13:49:16.282713 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.282687 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79"] Oct 30 13:49:16.348705 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.348672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.348839 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.348805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4s5\" (UniqueName: \"kubernetes.io/projected/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-kube-api-access-jj4s5\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.348890 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.348849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.348890 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.348872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.348955 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.348932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-metrics-client-ca\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.349048 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.349029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.349081 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.349066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.349109 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.349085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-tmp\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.349139 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.349124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-grpc-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450504 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-metrics-client-ca\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450654 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450654 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450654 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-tmp\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450654 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-grpc-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450930 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450930 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4s5\" (UniqueName: \"kubernetes.io/projected/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-kube-api-access-jj4s5\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450930 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.450930 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.450827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.451201 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.451033 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-tmp\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.451434 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.451402 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-metrics-client-ca\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.453597 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.453569 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.453926 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.453902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.454035 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.454010 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.454123 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.454102 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.454243 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.454223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-grpc-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.454298 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.454282 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.477143 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.477112 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4s5\" (UniqueName: \"kubernetes.io/projected/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-kube-api-access-jj4s5\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.588697 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.588664 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:16.925299 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:16.925272 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79"] Oct 30 13:49:16.926263 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:16.926240 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99134c6b_8fa1_443d_ac56_a69a2bfcc9cb.slice/crio-8b6eba9bacbad529c4f410fd1ab650a33136ce0427059789abac5c9d646e7d83 WatchSource:0}: Error finding container 8b6eba9bacbad529c4f410fd1ab650a33136ce0427059789abac5c9d646e7d83: Status 404 returned error can't find the container with id 8b6eba9bacbad529c4f410fd1ab650a33136ce0427059789abac5c9d646e7d83 Oct 30 13:49:17.071456 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:17.071416 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsrlt" event={"ID":"d413c8fe-be32-46db-9326-5cfb42fc987b","Type":"ContainerStarted","Data":"8d6d46de223a4eabd488fd25815bf3195f16ef3523d57032eb22cf868f14a2ca"} Oct 30 13:49:17.071665 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:17.071486 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsrlt" event={"ID":"d413c8fe-be32-46db-9326-5cfb42fc987b","Type":"ContainerStarted","Data":"de28766a65f7e7c10ec2247ddc7bc519fc2a11085e87e8b93e8ad59f763511bb"} Oct 30 13:49:17.072939 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:17.072908 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" event={"ID":"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb","Type":"ContainerStarted","Data":"8b6eba9bacbad529c4f410fd1ab650a33136ce0427059789abac5c9d646e7d83"} Oct 30 13:49:17.074273 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:17.074249 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerStarted","Data":"c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35"} Oct 30 13:49:17.119050 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:17.119000 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dsrlt" podStartSLOduration=3.022478608 podStartE2EDuration="4.118987557s" totalImagesPullingTime="1.016447263s" totalInitContainerRuntime="80.061686ms" isStatefulPod=true podCreationTimestamp="2026-10-30 13:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:17.118802475 +0000 UTC m=+58.905160524" watchObservedRunningTime="2026-10-30 13:49:17.118987557 +0000 UTC m=+58.905345606" Oct 30 13:49:18.079286 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.079248 2579 generic.go:356] "Generic (PLEG): container finished" podID="dc254d0c-5015-4683-98fa-c423c989b20c" containerID="3f03b57551d6d4f6f439ad4eac95479e996327e2b3a62befbb59e16ec6aa7551" exitCode=0 Oct 30 13:49:18.079765 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.079350 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerDied","Data":"3f03b57551d6d4f6f439ad4eac95479e996327e2b3a62befbb59e16ec6aa7551"} Oct 30 13:49:18.648398 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.648363 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76d44865b4-8hgxr"] Oct 30 13:49:18.650804 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.650781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.673939 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.673906 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76d44865b4-8hgxr"] Oct 30 13:49:18.682227 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.682172 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 13:49:18.682342 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.682248 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 13:49:18.682342 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.682293 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-f6xjc\"" Oct 30 13:49:18.685178 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.685156 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 13:49:18.685301 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.685222 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 13:49:18.685301 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.685225 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a76qod3b0ca7e\"" Oct 30 13:49:18.769982 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.769954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9d35bccc-a939-4913-8903-63b51885cc9a-audit-log\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.770116 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.769988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-client-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.770211 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.770118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-metrics-server-audit-profiles\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.770211 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.770156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhpr\" (UniqueName: \"kubernetes.io/projected/9d35bccc-a939-4913-8903-63b51885cc9a-kube-api-access-6xhpr\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.770315 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.770267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.770315 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.770303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-tls\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.770416 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.770395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-client-certs\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.871023 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.870976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhpr\" (UniqueName: \"kubernetes.io/projected/9d35bccc-a939-4913-8903-63b51885cc9a-kube-api-access-6xhpr\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.871369 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.871043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.871369 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.871083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-tls\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.871369 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.871140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-client-certs\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.871369 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.871185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9d35bccc-a939-4913-8903-63b51885cc9a-audit-log\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.871369 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.871234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-client-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.871369 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.871288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-metrics-server-audit-profiles\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.872042 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.871945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.872221 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.872173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9d35bccc-a939-4913-8903-63b51885cc9a-audit-log\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.872342 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.872305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-metrics-server-audit-profiles\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.874372 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.874334 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-client-certs\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.875229 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.875203 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-tls\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.876087 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.876063 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-client-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.909682 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.909605 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhpr\" (UniqueName: \"kubernetes.io/projected/9d35bccc-a939-4913-8903-63b51885cc9a-kube-api-access-6xhpr\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.963467 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.963420 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:18.982235 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.981110 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:18.982235 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.981154 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:18.989339 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:18.989318 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:19.086856 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:19.086780 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" event={"ID":"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb","Type":"ContainerStarted","Data":"13c340348fdf8f866cfacc0faf6dd37256e634a774a76c60155ac480b0cd8ccd"} Oct 30 13:49:19.092478 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:19.092291 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:19.291976 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:19.291942 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76d44865b4-8hgxr"] Oct 30 13:49:20.037707 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.037679 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:20.092694 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.092628 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" event={"ID":"9d35bccc-a939-4913-8903-63b51885cc9a","Type":"ContainerStarted","Data":"a9a3dce8249250a3aa57dcd5d6783b96305444d330df8a2c700b8f057a0e8041"} Oct 30 13:49:20.097417 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.097305 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" event={"ID":"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb","Type":"ContainerStarted","Data":"fa361d19718a0d7ee4f0a2ff4b89f628f744442eedd625d7d9f472d356f549ef"} Oct 30 13:49:20.097417 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.097384 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" event={"ID":"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb","Type":"ContainerStarted","Data":"7cf207ecd4bc0708d6e1ade8ccde7a04f6cd2c5949ce4e1624561eac9f331025"} Oct 30 13:49:20.101343 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.101311 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerStarted","Data":"e2d39a27ba075372648efc479afe802e20ff959aff3bf5479196010a065a141a"} Oct 30 13:49:20.101432 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.101383 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerStarted","Data":"0d90113eda2a6523354a9be7c6a641a0c5cb2942634c809dde2782da5e3c9fca"} Oct 30 13:49:20.101432 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.101422 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerStarted","Data":"de35cd767d5c4c5beee283e34ddc5072a27b59012cca6f940dc013bec5a4d201"} Oct 30 13:49:20.101542 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.101452 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerStarted","Data":"6865aa4d04bdc0cf06400509ee22c9e087684bb2e160896add23c3a1a6255dd3"} Oct 30 13:49:20.101542 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:20.101475 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerStarted","Data":"9885dc8de5dbb15bdd5d950bec67a2bc7702eb3c391c03c4f8cdeda65a2cc1e0"} Oct 30 13:49:21.073792 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.073702 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:21.077996 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.077971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.101834 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.101796 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 13:49:21.101834 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.101819 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 13:49:21.102396 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.102376 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 13:49:21.102833 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.102817 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fvfjd\"" Oct 30 13:49:21.112230 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.112200 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" event={"ID":"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb","Type":"ContainerStarted","Data":"dc83b0e69be30d86b2402b3d3963fb7431be2a7e38e3d4781815fb101e0418f5"} Oct 30 13:49:21.112349 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.112289 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" event={"ID":"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb","Type":"ContainerStarted","Data":"dde1d5b3b7c2975b513f28de388f23035a2945bac7631a38f57aac4c472e664c"} Oct 30 13:49:21.112349 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.112328 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" event={"ID":"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb","Type":"ContainerStarted","Data":"4be952bb86283eed8ddb9f35e4bd1d5027a7deea99c06c5c14846c8711abe28e"} Oct 30 13:49:21.112441 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.112373 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:21.113064 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.113040 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 13:49:21.113159 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.113093 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 13:49:21.113159 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.113125 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 13:49:21.113285 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.113200 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-58vdnq5moqt0o\"" Oct 30 13:49:21.113355 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.113336 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 13:49:21.113410 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.113339 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 13:49:21.118209 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.118175 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerStarted","Data":"16e70dc35b33f1c481914b0f5597ed058b073e7751d19b541a89db370cacd620"} Oct 30 13:49:21.120458 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.120436 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" event={"ID":"9d35bccc-a939-4913-8903-63b51885cc9a","Type":"ContainerStarted","Data":"9d15bbe4b9e81689d60cff9a4daee45785c46ac232cb9edba80b53aea668246d"} Oct 30 13:49:21.127341 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.127315 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 13:49:21.127433 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.127395 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 13:49:21.127682 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.127430 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 13:49:21.130110 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.130097 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 13:49:21.130266 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.130254 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 13:49:21.142279 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.142254 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 13:49:21.143688 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.143664 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:21.192711 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.192677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.192847 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.192730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193028 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193102 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193322 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193391 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193447 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193447 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193601 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193674 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193732 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193732 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193835 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.193979 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.193954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.194115 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.194010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.194115 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.194078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.194253 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.194153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.194311 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.194255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.194368 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.194315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.194425 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.194379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvqn\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.225303 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.225256 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" podStartSLOduration=2.338637487 podStartE2EDuration="5.225243412s" totalImagesPullingTime="2.886605925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:21.223762497 +0000 UTC m=+63.010120565" watchObservedRunningTime="2026-10-30 13:49:21.225243412 +0000 UTC m=+63.011601461" Oct 30 13:49:21.271291 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.271239 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" podStartSLOduration=1.8794093680000001 podStartE2EDuration="3.271225124s" totalImagesPullingTime="1.391815756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:21.271058471 +0000 UTC m=+63.057416519" watchObservedRunningTime="2026-10-30 13:49:21.271225124 +0000 UTC m=+63.057583173" Oct 30 13:49:21.295413 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295413 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295595 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295595 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295730 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295780 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295855 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295896 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.295950 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296003 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296003 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296003 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.295997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvqn\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296140 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296407 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296144 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296407 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296407 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296407 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.296828 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.298819 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.296890 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.298819 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.297493 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.298819 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.297624 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.299034 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.298923 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.299541 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.299270 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.299886 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.299856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.299987 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.299948 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.300156 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.300134 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.300525 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.300504 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.300926 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.300902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.301246 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.301222 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.301246 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.301225 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.301392 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.301342 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.301392 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.301358 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.301724 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.301706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.302489 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.302458 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.330029 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.329968 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvqn\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.336959 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.336918 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.407525767 podStartE2EDuration="6.336904996s" totalImagesPullingTime="2.915469622s" totalInitContainerRuntime="13.909607ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:21.335760334 +0000 UTC m=+63.122118382" watchObservedRunningTime="2026-10-30 13:49:21.336904996 +0000 UTC m=+63.123263045" Oct 30 13:49:21.412098 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.412065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:21.751158 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:21.751127 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:21.751859 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:21.751833 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242e5d46_f768_4a1e_b7ce_2d5f56690421.slice/crio-d17352c9210a8082ed7e9dd9a94e7182c5e7029b1d35ac8033c928976e31b49d WatchSource:0}: Error finding container d17352c9210a8082ed7e9dd9a94e7182c5e7029b1d35ac8033c928976e31b49d: Status 404 returned error can't find the container with id d17352c9210a8082ed7e9dd9a94e7182c5e7029b1d35ac8033c928976e31b49d Oct 30 13:49:22.125330 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:22.125250 2579 generic.go:356] "Generic (PLEG): container finished" podID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerID="2c8f7899ec329fdbb67f00de00c7ca26809a829224a26248a51dbf69efaf3b46" exitCode=0 Oct 30 13:49:22.125662 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:22.125353 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerDied","Data":"2c8f7899ec329fdbb67f00de00c7ca26809a829224a26248a51dbf69efaf3b46"} Oct 30 13:49:22.125662 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:22.125418 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerStarted","Data":"d17352c9210a8082ed7e9dd9a94e7182c5e7029b1d35ac8033c928976e31b49d"} Oct 30 13:49:24.531055 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.531002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:24.531563 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.531128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:24.546366 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.546331 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 13:49:24.554209 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.554168 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:24.559818 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.559773 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 13:49:24.570847 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.570817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:24.632276 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.632238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:24.647453 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.647420 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 13:49:24.667706 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.667671 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 13:49:24.667706 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.667675 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smj8j\"" Oct 30 13:49:24.670521 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.670498 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w8shz\"" Oct 30 13:49:24.673571 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.673551 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:24.675806 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.675789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:24.676425 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.676392 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:24.953338 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.953309 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gw59s\"" Oct 30 13:49:24.958905 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.958888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:24.990183 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:24.990160 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pb5tm"] Oct 30 13:49:25.011288 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:25.011248 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67379085_0be6_451f_b409_4e47dd190877.slice/crio-5f9fe1318118b023d5bb827c00dbafd7ec783d1458e843252002d3bebb0eb48d WatchSource:0}: Error finding container 5f9fe1318118b023d5bb827c00dbafd7ec783d1458e843252002d3bebb0eb48d: Status 404 returned error can't find the container with id 5f9fe1318118b023d5bb827c00dbafd7ec783d1458e843252002d3bebb0eb48d Oct 30 13:49:25.012289 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:25.011768 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gwvzz"] Oct 30 13:49:25.138675 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:25.138643 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pb5tm" event={"ID":"1b37398e-d79b-4ced-a7b6-07409e2dbdcc","Type":"ContainerStarted","Data":"a7c7ddc7cc8d6e49ad41f1cf1bdd4d198d7a8c0529d22b1f83bfc28cd30e9a16"} Oct 30 13:49:25.139921 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:25.139890 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwvzz" event={"ID":"67379085-0be6-451f-b409-4e47dd190877","Type":"ContainerStarted","Data":"5f9fe1318118b023d5bb827c00dbafd7ec783d1458e843252002d3bebb0eb48d"} Oct 30 13:49:25.142347 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:25.142323 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerStarted","Data":"854eb720a368331ae27bd6d29a445263329bf81f2d8a60fae03ad20f473b1e38"} Oct 30 13:49:25.142443 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:25.142386 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerStarted","Data":"daa8f54616f2c0c1c7ec7a4b8e26be6240e9bfcb6d63b3108ce881c7bc9ad07c"} Oct 30 13:49:25.304032 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:25.303655 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ldf6b"] Oct 30 13:49:25.312329 ip-10-0-131-200 kubenswrapper[2579]: W1030 13:49:25.312302 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f75fa3f_6f80_4ea7_a2a1_205839a11f12.slice/crio-b00acad6d839a339f62aeb53821ac3b0120cfb18d6c81768f866dbb2100a9eeb WatchSource:0}: Error finding container b00acad6d839a339f62aeb53821ac3b0120cfb18d6c81768f866dbb2100a9eeb: Status 404 returned error can't find the container with id b00acad6d839a339f62aeb53821ac3b0120cfb18d6c81768f866dbb2100a9eeb Oct 30 13:49:26.149365 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:26.149234 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ldf6b" event={"ID":"1f75fa3f-6f80-4ea7-a2a1-205839a11f12","Type":"ContainerStarted","Data":"b00acad6d839a339f62aeb53821ac3b0120cfb18d6c81768f866dbb2100a9eeb"} Oct 30 13:49:26.153632 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:26.153599 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerStarted","Data":"488fd8a8e6459f9a67b9c370a0d3a5563f10946ff27a79184ff32a7b70382cc7"} Oct 30 13:49:26.153748 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:26.153675 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerStarted","Data":"8ca88eeb9f3a532167453bceed44eab8a331ed4fcd7eb250d673e67c6a934063"} Oct 30 13:49:26.153748 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:26.153715 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerStarted","Data":"9084cd1e0f4d6fc898f79c254acdca7060f8e50b63410cf8363f506248abdfb8"} Oct 30 13:49:26.153835 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:26.153751 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerStarted","Data":"10445525592b8f18be8424ef153e97cb243074b5992370fa8bf28a0af3d0ffe4"} Oct 30 13:49:26.227890 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:26.227832 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.378890022 podStartE2EDuration="5.227815574s" totalImagesPullingTime="2.831963702s" totalInitContainerRuntime="16.96185ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:26.22539281 +0000 UTC m=+68.011750895" watchObservedRunningTime="2026-10-30 13:49:26.227815574 +0000 UTC m=+68.014173622" Oct 30 13:49:26.412609 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:26.412581 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:27.132949 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:27.132921 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:27.163797 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:27.163763 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwvzz" event={"ID":"67379085-0be6-451f-b409-4e47dd190877","Type":"ContainerStarted","Data":"010e34f804986ffc060bd995c3a12f6d4262932d81481d12a28854c73cc9ba7c"} Oct 30 13:49:27.164221 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:27.163822 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwvzz" event={"ID":"67379085-0be6-451f-b409-4e47dd190877","Type":"ContainerStarted","Data":"d67977576740c431379ffc0839eac476709a61583b0fd4435daf5ca9b0fa28b2"} Oct 30 13:49:27.239865 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:27.239808 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gwvzz" podStartSLOduration=66.988285076 podStartE2EDuration="1m8.239791093s" totalImagesPullingTime="1.251506017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:27.238547638 +0000 UTC m=+69.024905686" watchObservedRunningTime="2026-10-30 13:49:27.239791093 +0000 UTC m=+69.026149143" Oct 30 13:49:29.173921 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:29.173879 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ldf6b" event={"ID":"1f75fa3f-6f80-4ea7-a2a1-205839a11f12","Type":"ContainerStarted","Data":"96d09e4dbc9d2aedc185107d9a51f032f1a6274fbce8cffab4f8b92dd28ecaef"} Oct 30 13:49:29.174318 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:29.174060 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:29.221929 ip-10-0-131-200 kubenswrapper[2579]: I1030 13:49:29.221859 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ldf6b" podStartSLOduration=66.816230424 podStartE2EDuration="1m10.221836775s" totalImagesPullingTime="3.405606351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:29.220043527 +0000 UTC m=+71.006401568" watchObservedRunningTime="2026-10-30 13:49:29.221836775 +0000 UTC m=+71.008194826" Oct 30 13:49:30.143180 ip-10-0-131-200 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 13:49:30.154837 ip-10-0-131-200 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 13:49:30.155015 ip-10-0-131-200 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 13:49:30.155074 ip-10-0-131-200 systemd[1]: kubelet.service: Consumed 4.374s CPU time. Oct 30 13:49:30.164839 ip-10-0-131-200 systemd[1]: Starting Kubernetes Kubelet... Oct 30 13:49:30.231610 ip-10-0-131-200 kubenswrapper[5824]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:49:30.231610 ip-10-0-131-200 kubenswrapper[5824]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 13:49:30.231610 ip-10-0-131-200 kubenswrapper[5824]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:49:30.231610 ip-10-0-131-200 kubenswrapper[5824]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 13:49:30.233737 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233644 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:49:30.233737 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233736 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233742 5824 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233747 5824 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233751 5824 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233753 5824 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233756 5824 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233767 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233770 5824 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233773 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233775 5824 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233780 5824 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233783 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233786 5824 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233789 5824 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233792 5824 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233795 5824 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233798 5824 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233801 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233803 5824 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:49:30.233818 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233807 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233809 5824 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233812 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233815 5824 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233818 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233821 5824 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233824 5824 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233827 5824 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233830 5824 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233833 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233836 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233839 5824 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233842 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233845 5824 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233848 5824 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233850 5824 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233853 5824 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233856 5824 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233858 5824 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:49:30.234354 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233861 5824 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233863 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233866 5824 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233869 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233872 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233875 5824 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233877 5824 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233880 5824 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233882 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233885 5824 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233888 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233890 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233893 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233895 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233898 5824 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233901 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233904 5824 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233907 5824 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233909 5824 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233912 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:49:30.234807 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233915 5824 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233918 5824 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233920 5824 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233923 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233926 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233929 5824 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233932 5824 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233934 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233937 5824 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233939 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233942 5824 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233945 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233948 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233951 5824 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233954 5824 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233956 5824 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233959 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233961 5824 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233964 5824 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:49:30.235326 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233966 5824 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233969 5824 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233972 5824 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233974 5824 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233977 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233979 5824 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233982 5824 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233984 5824 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233988 5824 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233991 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233993 5824 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233996 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.233999 5824 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234001 5824 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234004 5824 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234007 5824 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234010 5824 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234013 5824 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234016 5824 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234018 5824 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:49:30.235780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234021 5824 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234024 5824 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234027 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234029 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234032 5824 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234034 5824 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234037 5824 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234040 5824 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234865 5824 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234873 5824 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234877 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234880 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234883 5824 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234888 5824 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234892 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234895 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234899 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234901 5824 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234904 5824 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:49:30.236280 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234907 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234915 5824 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234918 5824 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234921 5824 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234923 5824 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234926 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234929 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234932 5824 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234934 5824 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234937 5824 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234940 5824 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234942 5824 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234945 5824 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234948 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234951 5824 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234954 5824 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234957 5824 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234959 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234962 5824 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:49:30.236743 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234965 5824 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234967 5824 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234970 5824 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234972 5824 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234975 5824 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234977 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234980 5824 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234982 5824 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234985 5824 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234988 5824 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234990 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234993 5824 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234995 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.234998 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235003 5824 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235006 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235009 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235011 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235014 5824 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235016 5824 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:49:30.237233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235019 5824 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235022 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235024 5824 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235028 5824 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235032 5824 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235035 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235039 5824 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235042 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235045 5824 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235048 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235051 5824 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235053 5824 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235056 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235058 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235061 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235063 5824 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235066 5824 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235069 5824 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235071 5824 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:49:30.237726 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235074 5824 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235076 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235079 5824 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235081 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235084 5824 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235087 5824 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235089 5824 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235096 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235098 5824 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235101 5824 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235104 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235106 5824 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235109 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235112 5824 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235114 5824 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235117 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235120 5824 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235122 5824 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235125 5824 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235136 5824 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:49:30.238233 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235139 5824 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235142 5824 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235145 5824 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235148 5824 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235151 5824 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235153 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235156 5824 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235159 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235161 5824 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235164 5824 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235167 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235169 5824 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235172 5824 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235174 5824 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235177 5824 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235180 5824 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235182 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235684 5824 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235690 5824 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235696 5824 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:49:30.238727 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235699 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235702 5824 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235705 5824 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235708 5824 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235710 5824 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235713 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235716 5824 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235719 5824 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235721 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235724 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235727 5824 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235729 5824 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235732 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235735 5824 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235738 5824 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235741 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235744 5824 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235746 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235749 5824 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235752 5824 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:49:30.239232 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235754 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235757 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235759 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235762 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235764 5824 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235767 5824 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235770 5824 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235772 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235774 5824 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235777 5824 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235779 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235786 5824 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235789 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235792 5824 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235795 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235797 5824 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235800 5824 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235804 5824 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235808 5824 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235811 5824 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:49:30.239780 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235813 5824 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235816 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235819 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235821 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235824 5824 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235826 5824 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235829 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235832 5824 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235835 5824 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235837 5824 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235840 5824 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235842 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235845 5824 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235847 5824 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235850 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235852 5824 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235855 5824 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235857 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235860 5824 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235862 5824 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:49:30.240347 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235865 5824 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235867 5824 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235870 5824 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235876 5824 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235879 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235881 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235884 5824 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235887 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235889 5824 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235892 5824 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235894 5824 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235897 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235899 5824 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235902 5824 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235905 5824 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235908 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235910 5824 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235913 5824 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235915 5824 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:49:30.241147 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235918 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235921 5824 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235924 5824 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235926 5824 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235929 5824 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235931 5824 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235934 5824 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235936 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235939 5824 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235941 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235944 5824 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235946 5824 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235949 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235951 5824 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235954 5824 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235956 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235961 5824 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235964 5824 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235966 5824 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:49:30.241675 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235969 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235971 5824 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235974 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235980 5824 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.235984 5824 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236079 5824 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236087 5824 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236094 5824 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236099 5824 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236104 5824 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236107 5824 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236114 5824 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236119 5824 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236122 5824 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236126 5824 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236130 5824 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236134 5824 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236137 5824 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236141 5824 flags.go:64] FLAG: --cgroup-root="" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236144 5824 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236147 5824 flags.go:64] FLAG: --client-ca-file="" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236150 5824 flags.go:64] FLAG: --cloud-provider="external" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236153 5824 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 13:49:30.242144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236157 5824 flags.go:64] FLAG: --cluster-domain="" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236160 5824 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236164 5824 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236167 5824 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236170 5824 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236174 5824 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236179 5824 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236182 5824 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236200 5824 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236205 5824 flags.go:64] FLAG: --contention-profiling="false" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236208 5824 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236211 5824 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236215 5824 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236218 5824 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236223 5824 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236226 5824 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236229 5824 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236232 5824 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236235 5824 flags.go:64] FLAG: --enable-server="true" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236239 5824 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236244 5824 flags.go:64] FLAG: --event-burst="100" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236247 5824 flags.go:64] FLAG: --event-qps="50" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236250 5824 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236254 5824 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 13:49:30.242724 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236257 5824 flags.go:64] FLAG: --eviction-hard="" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236262 5824 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236266 5824 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236269 5824 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236272 5824 flags.go:64] FLAG: --eviction-soft="" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236275 5824 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236279 5824 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236282 5824 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236285 5824 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236288 5824 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236291 5824 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236294 5824 flags.go:64] FLAG: --feature-gates="" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236300 5824 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236303 5824 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236307 5824 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236310 5824 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236313 5824 flags.go:64] FLAG: --healthz-port="10248" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236316 5824 flags.go:64] FLAG: --help="false" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236319 5824 flags.go:64] FLAG: --hostname-override="ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236323 5824 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236326 5824 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236328 5824 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236332 5824 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 13:49:30.243334 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236335 5824 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236338 5824 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236341 5824 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236344 5824 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236347 5824 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236350 5824 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236353 5824 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236356 5824 flags.go:64] FLAG: --kube-reserved="" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236361 5824 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236365 5824 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236368 5824 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236371 5824 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236374 5824 flags.go:64] FLAG: --lock-file="" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236377 5824 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236380 5824 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236383 5824 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236389 5824 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236392 5824 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236395 5824 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236397 5824 flags.go:64] FLAG: --logging-format="text" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236400 5824 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236404 5824 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236407 5824 flags.go:64] FLAG: --manifest-url="" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236410 5824 flags.go:64] FLAG: --manifest-url-header="" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236415 5824 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 13:49:30.243895 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236418 5824 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236422 5824 flags.go:64] FLAG: --max-pods="110" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236425 5824 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236428 5824 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236431 5824 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236435 5824 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236438 5824 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236441 5824 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236444 5824 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236451 5824 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236454 5824 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236458 5824 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236461 5824 flags.go:64] FLAG: --pod-cidr="" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236464 5824 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236467 5824 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236471 5824 flags.go:64] FLAG: --pods-per-core="0" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236475 5824 flags.go:64] FLAG: --port="10250" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236477 5824 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236481 5824 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f6456afe4bcccee" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236487 5824 flags.go:64] FLAG: --qos-reserved="" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236491 5824 flags.go:64] FLAG: --read-only-port="10255" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236494 5824 flags.go:64] FLAG: --register-node="true" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236497 5824 flags.go:64] FLAG: --register-with-taints="" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236501 5824 flags.go:64] FLAG: --registry-burst="10" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236504 5824 flags.go:64] FLAG: --registry-qps="5" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236508 5824 flags.go:64] FLAG: --reserved-cpus="" Oct 30 13:49:30.244524 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236511 5824 flags.go:64] FLAG: --reserved-memory="" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236515 5824 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236519 5824 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236522 5824 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236525 5824 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236528 5824 flags.go:64] FLAG: --runonce="false" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236531 5824 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236534 5824 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236537 5824 flags.go:64] FLAG: --seccomp-default="false" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236540 5824 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236543 5824 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236546 5824 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236549 5824 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236552 5824 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236555 5824 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236558 5824 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236561 5824 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236564 5824 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236567 5824 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236570 5824 flags.go:64] FLAG: --system-cgroups="" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236573 5824 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236580 5824 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236583 5824 flags.go:64] FLAG: --tls-cert-file="" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236586 5824 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236590 5824 flags.go:64] FLAG: --tls-min-version="" Oct 30 13:49:30.245239 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236593 5824 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236597 5824 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236600 5824 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236603 5824 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236606 5824 flags.go:64] FLAG: --v="2" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236611 5824 flags.go:64] FLAG: --version="false" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236615 5824 flags.go:64] FLAG: --vmodule="" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236619 5824 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.236623 5824 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236759 5824 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236763 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236766 5824 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236769 5824 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236772 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236774 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236777 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236779 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236783 5824 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236785 5824 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236788 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236790 5824 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:49:30.245846 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236793 5824 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236796 5824 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236798 5824 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236801 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236804 5824 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236806 5824 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236809 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236813 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236817 5824 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236822 5824 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236825 5824 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236828 5824 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236831 5824 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236834 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236837 5824 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236839 5824 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236842 5824 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236848 5824 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236851 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:49:30.246364 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236854 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236856 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236859 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236862 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236864 5824 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236867 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236869 5824 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236872 5824 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236875 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236877 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236880 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236883 5824 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236885 5824 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236888 5824 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236890 5824 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236893 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236895 5824 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236898 5824 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236901 5824 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236903 5824 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:49:30.246855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236908 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236911 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236915 5824 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236917 5824 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236920 5824 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236923 5824 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236926 5824 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236929 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236931 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236934 5824 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236937 5824 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236940 5824 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236943 5824 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236945 5824 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236948 5824 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236950 5824 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236953 5824 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236956 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236958 5824 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:49:30.247559 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236961 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236963 5824 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236966 5824 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236968 5824 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236971 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236974 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236976 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236979 5824 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236982 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236984 5824 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236987 5824 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236990 5824 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236992 5824 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.236997 5824 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237001 5824 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237005 5824 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237009 5824 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237011 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237014 5824 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237017 5824 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:49:30.248226 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237020 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237023 5824 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237025 5824 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237028 5824 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237031 5824 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237033 5824 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237036 5824 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237039 5824 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237041 5824 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237044 5824 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237046 5824 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237049 5824 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237052 5824 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237054 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237057 5824 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237060 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:49:30.248889 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.237085 5824 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237123 5824 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237129 5824 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237132 5824 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237135 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237138 5824 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237140 5824 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237143 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237146 5824 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237150 5824 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237153 5824 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237156 5824 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237160 5824 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237162 5824 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237165 5824 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237168 5824 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237171 5824 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237173 5824 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237176 5824 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:49:30.249370 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237179 5824 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237182 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237184 5824 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237204 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237209 5824 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237212 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237217 5824 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237220 5824 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237224 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237228 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237232 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237236 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237241 5824 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237245 5824 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237248 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237251 5824 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237255 5824 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237257 5824 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237260 5824 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:49:30.249855 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237263 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237266 5824 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237268 5824 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237273 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237276 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237281 5824 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237287 5824 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237290 5824 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237293 5824 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237296 5824 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237299 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237302 5824 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237304 5824 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237307 5824 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237310 5824 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237313 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237316 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237319 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237321 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:49:30.250498 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237324 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237327 5824 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237330 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237332 5824 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237335 5824 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237337 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237340 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237343 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237345 5824 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237348 5824 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237351 5824 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237354 5824 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237356 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237359 5824 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237362 5824 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237364 5824 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237369 5824 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237372 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237375 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237379 5824 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:49:30.251143 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237381 5824 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237384 5824 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237387 5824 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237389 5824 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237392 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237395 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237397 5824 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237401 5824 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237404 5824 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237407 5824 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237409 5824 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237412 5824 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237415 5824 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237417 5824 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237420 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237423 5824 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237425 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237428 5824 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237431 5824 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:49:30.251785 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237433 5824 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237436 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237439 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237441 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237444 5824 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237447 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237449 5824 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237453 5824 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237456 5824 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237461 5824 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.237463 5824 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.242384 5824 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.242404 5824 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.242425 5824 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.242430 5824 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242492 5824 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242498 5824 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242501 5824 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242505 5824 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242508 5824 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:49:30.252375 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242511 5824 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242514 5824 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242517 5824 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242520 5824 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242523 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242526 5824 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242529 5824 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242531 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242534 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242542 5824 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242545 5824 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242548 5824 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242551 5824 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242554 5824 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242557 5824 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242559 5824 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242562 5824 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242565 5824 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242567 5824 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242570 5824 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:49:30.252896 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242573 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242576 5824 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242579 5824 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242581 5824 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242584 5824 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242587 5824 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242590 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242593 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242596 5824 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242598 5824 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242601 5824 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242604 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242606 5824 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242609 5824 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242611 5824 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242614 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242617 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242620 5824 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242623 5824 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242625 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:49:30.253429 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242628 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242631 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242634 5824 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242636 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242639 5824 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242641 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242645 5824 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242647 5824 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242650 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242652 5824 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242656 5824 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242660 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242664 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242667 5824 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242670 5824 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242673 5824 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242676 5824 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242679 5824 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242682 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:49:30.254169 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242685 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242688 5824 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242690 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242693 5824 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242696 5824 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242699 5824 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242701 5824 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242704 5824 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242706 5824 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242709 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242711 5824 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242714 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242716 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242719 5824 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242722 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242725 5824 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242728 5824 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242731 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242734 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:49:30.255213 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242737 5824 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242740 5824 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242745 5824 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242749 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242752 5824 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242754 5824 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242757 5824 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242760 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242762 5824 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242765 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242768 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242771 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242773 5824 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242776 5824 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242778 5824 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242781 5824 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242784 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242786 5824 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242789 5824 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:49:30.256061 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242792 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242795 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242797 5824 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242800 5824 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.242830 5824 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242989 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242994 5824 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.242998 5824 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243001 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243004 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243008 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243010 5824 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243013 5824 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243016 5824 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243019 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243021 5824 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:49:30.256689 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243024 5824 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243027 5824 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243029 5824 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243032 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243035 5824 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243037 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243040 5824 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243042 5824 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243045 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243047 5824 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243050 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243053 5824 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243056 5824 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243059 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243061 5824 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243064 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243067 5824 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243069 5824 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243072 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:49:30.257086 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243075 5824 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243078 5824 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243081 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243084 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243087 5824 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243089 5824 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243092 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243094 5824 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243098 5824 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243101 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243104 5824 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243107 5824 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243109 5824 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243112 5824 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243114 5824 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243117 5824 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243119 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243122 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243125 5824 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243136 5824 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:49:30.257558 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243139 5824 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243142 5824 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243144 5824 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243147 5824 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243150 5824 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243153 5824 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243155 5824 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243158 5824 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243176 5824 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243181 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243184 5824 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243200 5824 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243203 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243206 5824 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243209 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243212 5824 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243214 5824 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243217 5824 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243220 5824 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243222 5824 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:49:30.258044 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243225 5824 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243228 5824 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243231 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243234 5824 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243237 5824 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243239 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243242 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243245 5824 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243247 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243250 5824 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243253 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243255 5824 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243259 5824 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243263 5824 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243266 5824 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243268 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243271 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243274 5824 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243276 5824 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:49:30.258627 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243279 5824 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243285 5824 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243288 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243290 5824 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243293 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243295 5824 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243298 5824 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243301 5824 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243303 5824 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243306 5824 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243309 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243313 5824 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243316 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243320 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243322 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243326 5824 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:49:30.259113 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243330 5824 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.243354 5824 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243388 5824 feature_gate.go:386] unrecognized feature gate: Example Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243393 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243396 5824 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243399 5824 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243402 5824 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243405 5824 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243408 5824 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243411 5824 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243414 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243416 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243419 5824 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243421 5824 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243424 5824 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243427 5824 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243429 5824 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 13:49:30.259804 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243432 5824 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243435 5824 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243438 5824 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243440 5824 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243443 5824 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243446 5824 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243448 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243451 5824 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243454 5824 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243456 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243459 5824 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243461 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243463 5824 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243466 5824 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243469 5824 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243473 5824 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243475 5824 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243478 5824 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243480 5824 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243483 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 13:49:30.260522 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243486 5824 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243488 5824 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243491 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243494 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243496 5824 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243499 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243502 5824 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243504 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243507 5824 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243510 5824 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243514 5824 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243517 5824 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243519 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243522 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243525 5824 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243528 5824 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243531 5824 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243533 5824 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243536 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243538 5824 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 13:49:30.261137 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243541 5824 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243543 5824 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243546 5824 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243548 5824 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243551 5824 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243554 5824 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243557 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243561 5824 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243565 5824 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243568 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243571 5824 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243573 5824 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243576 5824 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243578 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243581 5824 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243583 5824 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243586 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243588 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243591 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 13:49:30.261991 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243593 5824 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243596 5824 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243598 5824 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243601 5824 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243604 5824 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243607 5824 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243609 5824 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243612 5824 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243615 5824 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243617 5824 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243620 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243622 5824 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243625 5824 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243627 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243630 5824 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243632 5824 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243635 5824 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243637 5824 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243640 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243645 5824 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 13:49:30.262760 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243648 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243650 5824 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243654 5824 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243656 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243659 5824 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243662 5824 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243664 5824 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243666 5824 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243669 5824 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243672 5824 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243675 5824 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:30.243677 5824 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.243745 5824 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.244764 5824 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.244839 5824 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.245047 5824 server.go:1014] "Starting client certificate rotation" Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.245202 5824 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 13:49:30.263485 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.245256 5824 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 13:43:19 +0000 UTC" deadline="2028-04-15 23:54:04.763547209 +0000 UTC" Oct 30 13:49:30.263920 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.245285 5824 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12802h4m34.51826422s" Oct 30 13:49:30.263920 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.245597 5824 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:49:30.263920 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.246791 5824 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 13:49:30.263920 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.248073 5824 log.go:25] "Validated CRI v1 runtime API" Oct 30 13:49:30.263920 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.250124 5824 log.go:25] "Validated CRI v1 image API" Oct 30 13:49:30.263920 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.251472 5824 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 13:49:30.263920 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.254248 5824 fs.go:135] Filesystem UUIDs: map[157e2c80-860f-4a35-be8e-94ada3792988:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b1a33dc4-1384-4315-9baf-386c7ee93c21:/dev/nvme0n1p4] Oct 30 13:49:30.264129 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.254278 5824 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/171b1fcd87d07a1dd74252030a4a51cb0af23e11ba6bdd5b097fd6fd086c634a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/171b1fcd87d07a1dd74252030a4a51cb0af23e11ba6bdd5b097fd6fd086c634a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c116049c5f0051c4c255c94f95aebaa27a033149b9dbe649dcd99b2b3370f01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c116049c5f0051c4c255c94f95aebaa27a033149b9dbe649dcd99b2b3370f01/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3df94a1d75c19881b6612fb5902ea860d3574702a8f4d289f07f476189b2c655/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3df94a1d75c19881b6612fb5902ea860d3574702a8f4d289f07f476189b2c655/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/461354a6ffb262a94623241533d908493ffd3c85de2a91399503f640414b7a49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/461354a6ffb262a94623241533d908493ffd3c85de2a91399503f640414b7a49/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ff2a8b95f6634d328e971d1c6be6484e6c41c18d3e2fc269be006d38c54a004/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ff2a8b95f6634d328e971d1c6be6484e6c41c18d3e2fc269be006d38c54a004/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5121026b463798d4c27877cdf06084756d832e3ac9e6032893c057e9b93652e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5121026b463798d4c27877cdf06084756d832e3ac9e6032893c057e9b93652e3/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/585e43fbe5f67219d4bb3d45bd9564d3dc8a604f4e21318542aabdbd110744b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/585e43fbe5f67219d4bb3d45bd9564d3dc8a604f4e21318542aabdbd110744b5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5940df8af7d84bf7c51ed70150627b5a0da2fa549fd13318dcbb2e7c63377875/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5940df8af7d84bf7c51ed70150627b5a0da2fa549fd13318dcbb2e7c63377875/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f9fe1318118b023d5bb827c00dbafd7ec783d1458e843252002d3bebb0eb48d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f9fe1318118b023d5bb827c00dbafd7ec783d1458e843252002d3bebb0eb48d/userdata/shm major:0 minor:441 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b6eba9bacbad529c4f410fd1ab650a33136ce0427059789abac5c9d646e7d83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b6eba9bacbad529c4f410fd1ab650a33136ce0427059789abac5c9d646e7d83/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a187132a53451c593de5a474312eb47b6f79e64d2d0fd04e87c0c58567261b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a187132a53451c593de5a474312eb47b6f79e64d2d0fd04e87c0c58567261b9/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e0458ec69773388dca1845aea143308502241a12b9c8d0981f677f9e946638b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e0458ec69773388dca1845aea143308502241a12b9c8d0981f677f9e946638b/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e08d19cc1d810131e0b376a84681647f90730ea81e2386a07e1e42538441af5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e08d19cc1d810131e0b376a84681647f90730ea81e2386a07e1e42538441af5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a55fa4712649caf1ac5b901febddbac3b15244d680f35749e25e8405edfbefea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a55fa4712649caf1ac5b901febddbac3b15244d680f35749e25e8405edfbefea/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7c7ddc7cc8d6e49ad41f1cf1bdd4d198d7a8c0529d22b1f83bfc28cd30e9a16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7c7ddc7cc8d6e49ad41f1cf1bdd4d198d7a8c0529d22b1f83bfc28cd30e9a16/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9a3dce8249250a3aa57dcd5d6783b96305444d330df8a2c700b8f057a0e8041/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9a3dce8249250a3aa57dcd5d6783b96305444d330df8a2c700b8f057a0e8041/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afb9afd21cc05977b91a894bf16e7f1c413489d3ec7e2a24deb753a3396e0e35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afb9afd21cc05977b91a894bf16e7f1c413489d3ec7e2a24deb753a3396e0e35/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b00acad6d839a339f62aeb53821ac3b0120cfb18d6c81768f866dbb2100a9eeb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b00acad6d839a339f62aeb53821ac3b0120cfb18d6c81768f866dbb2100a9eeb/userdata/shm major:0 minor:445 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7aac1d6b69598fa39e2a64d5614ca1c7e2f159028be83206d735501d5477508/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7aac1d6b69598fa39e2a64d5614ca1c7e2f159028be83206d735501d5477508/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35/userdata/shm major:0 minor:313 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d17352c9210a8082ed7e9dd9a94e7182c5e7029b1d35ac8033c928976e31b49d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d17352c9210a8082ed7e9dd9a94e7182c5e7029b1d35ac8033c928976e31b49d/userdata/shm major:0 minor:434 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8e3c753d2c66acbeeb30d790d7b514dd5ddc05651808bbdfad823c0cf77f1df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8e3c753d2c66acbeeb30d790d7b514dd5ddc05651808bbdfad823c0cf77f1df/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc933f491eac7b20ed2e7a31879bdc18af87108dda08a83857cf02534067f187/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc933f491eac7b20ed2e7a31879bdc18af87108dda08a83857cf02534067f187/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6ddd1f9a5297f1b2b31cabbe78e24dcc066e3311f6b515783eea7abcd28170b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6ddd1f9a5297f1b2b31cabbe78e24dcc066e3311f6b515783eea7abcd28170b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~projected/kube-api-access-9mshk:{mountpoint:/var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~projected/kube-api-access-9mshk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b37398e-d79b-4ced-a7b6-07409e2dbdcc/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1b37398e-d79b-4ced-a7b6-07409e2dbdcc/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f75fa3f-6f80-4ea7-a2a1-205839a11f12/volumes/kubernetes.io~projected/kube-api-access-p82w4:{mountpoint:/var/lib/kubelet/pods/1f75fa3f-6f80-4ea7-a2a1-205839a11f12/volumes/kubernetes.io~projected/kube-api-access-p82w4 major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/config-reloader/2 major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/prometheus/15 major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~empty-dir/config-out major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~projected/kube-api-access-xdvqn:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~projected/kube-api-access-xdvqn major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~projected/tls-assets major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/config major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/web-config major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~projected/kube-api-access-xtp2p:{mountpoint:/var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~projected/kube-api-access-xtp2p major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cdb85a7-7e6c-4e60-bbac-dbf902150c60/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5cdb85a7-7e6c-4e60-bbac-dbf902150c60/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67379085-0be6-451f-b409-4e47dd190877/volumes/kubernetes.io~projected/kube-api-access-dpq4l:{mountpoint:/var/lib/kubelet/pods/67379085-0be6-451f-b409-4e47dd190877/volumes/kubernetes.io~projected/kube-api-access-dpq4l major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67379085-0be6-451f-b409-4e47dd190877/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/67379085-0be6-451f-b409-4e47dd190877/volumes/kubernetes.io~secret/metrics-certs major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72a2c31f-6357-4b42-be58-9672ec8dcecb/volumes/kubernetes.io~projected/kube-api-access-dtsfg:{mountpoint:/var/lib/kubelet/pods/72a2c31f-6357-4b42-be58-9672ec8dcecb/volumes/kubernetes.io~projected/kube-api-access-dtsfg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e9025a0-9daa-426a-b18d-3a04a154b693/volumes/kubernetes.io~projected/kube-api-access-qjs8f:{mountpoint:/var/lib/kubelet/pods/7e9025a0-9daa-426a-b18d-3a04a154b693/volumes/kubernetes.io~projected/kube-api-access-qjs8f major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~projected/kube-api-access-qw7ft:{mountpoint:/var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~projected/kube-api-access-qw7ft major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~projected/kube-api-access-jj4s5:{mountpoint:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~projected/kube-api-access-jj4s5 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~projected/kube-api-access-6xhpr:{mountpoint:/var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~projected/kube-api-access-6xhpr major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a40f373a-ac17-46aa-affb-41fc629af3da/volumes/kubernetes.io~projected/kube-api-access-ft2d8:{mountpoint:/var/lib/kubelet/pods/a40f373a-ac17-46aa-affb-41fc629af3da/volumes/kubernetes.io~projected/kube-api-access-ft2d8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a53bb1dd-634e-4c2f-9e71-306c2136c837/volumes/kubernetes.io~projected/kube-api-access-4h4wd:{mountpoint:/var/lib/kubelet/pods/a53bb1dd-634e-4c2f-9e71-306c2136c837/volumes/kubernetes.io~projected/kube-api-access-4h4wd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c082c71c-c2af-4b9d-bde3-35e11d42fb51/volumes/kubernetes.io~projected/kube-api-access-s5qvh:{mountpoint:/var/lib/kubelet/pods/c082c71c-c2af-4b9d-bde3-35e11d42fb51/volumes/kubernetes.io~projected/kube-api-access-s5qvh major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c082c71c-c2af-4b9d-bde3-35e11d42fb51/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c082c71c-c2af-4b9d-bde3-35e11d42fb51/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c59f3642-4311-4da7-b3e2-66c782e2b83b/volumes/kubernetes.io~projected/kube-api-access-ggnrl:{mountpoint:/var/lib/kubelet/pods/c59f3642-4311-4da7-b3e2-66c782e2b83b/volumes/kubernetes.io~projected/kube-api-access-ggnrl major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c80b9b41-b573-4246-b0ca-fd5e7297e6f0/volumes/kubernetes.io~projected/kube-api-access-dsvz6:{mountpoint:/var/lib/kubelet/pods/c80b9b41-b573-4246-b0ca-fd5e7297e6f0/volumes/kubernetes.io~projected/kube-api-access-dsvz6 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c80b9b41-b573-4246-b0ca-fd5e7297e6f0/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c80b9b41-b573-4246-b0ca-fd5e7297e6f0/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~projected/kube-api-access-gcwsv:{mountpoint:/var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~projected/kube-api-access-gcwsv major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/alertmanager/9 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/config-reloader/6 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~projected/kube-api-access-szdqr:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~projected/kube-api-access-szdqr major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/config-volume major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/web-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~projected/kube-api-access-tm9fg:{mountpoint:/var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~projected/kube-api-access-tm9fg major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9f9b8de-895f-4e27-b089-0f97cfefdff8/volumes/kubernetes.io~projected/kube-api-access-cmqsp:{mountpoint:/var/lib/kubelet/pods/e9f9b8de-895f-4e27-b089-0f97cfefdff8/volumes/kubernetes.io~projected/kube-api-access-cmqsp major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9f9b8de-895f-4e27-b089-0f97cfefdff8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e9f9b8de-895f-4e27-b089-0f97cfefdff8/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11ccc81-ad15-4901-afe8-dbc49587fb90/volumes/kubernetes.io~projected/kube-api-access-9xhn2:{mountpoint:/var/lib/kubelet/pods/f11ccc81-ad15-4901-afe8-dbc49587fb90/volumes/kubernetes.io~projected/kube-api-access-9xhn2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volumes/kubernetes.io~projected/kube-api-access-4gjnm:{mountpoint:/var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volumes/kubernetes.io~projected/kube-api-access-4gjnm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c9c7cf065e5b735c02a53c3da406a4004a34f3b7f5a8878a1df97f395c765e07/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a2f36d82bf8775c05c0715a538eaf325c306ef125473d1b297100666a94770f5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/75330b07499fe3467cc9b9f028b75cc72174c1e504ccc655947ffab7e89c8425/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/5b1d2450c14d4ddef554ba8d5fdc3521f1b008a51981508a4500759db2e87552/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d969a877bfd742aada47fe004eef853dc5c5ac7e184028834ab8f0670ecf61bd/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4e5c5d0dfcb4d1ae980e7ad7f734197ab8822631d78910d200a385b846bf923f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a2638cb224c309e4505180758e892a60e6ba39eda94d82258c93848bc8ee0102/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ac6525cc7b0eed0f31a8a66c3a7f26fc795ab0e269e2fc3b36046ba9181fec45/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/36d2c7f3cacc5ae4d0dbbc1c65198c386f660041762dbca816826d6c76ae0f5f/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8b992eff22517767693cf10ca603501c3a59127b06f9efe83b5baa457f6a8ee2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/3820d15385ff6c1c452328342170a58653e5cb6f63ff4c183e9b8c178f629b92/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/79c9399754c688abe578d99bc96c7fe521322b1fcde3a4cb9732f201878bd6b1/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/f3f86003c18e53842d0069d2ed483e4d5583ea9aa454313ff14def5c0636b2c9/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c2b0b7bf32d18e28a09dbeb0c93f08e129280483745208ba93394920118a7672/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8e42e952c90f5e258f906f521b55bb42a2ee9798f45d7269086b39b51d868e35/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/62185d7d861e99cc5ae2b3815ad2e5b5c79542eec01c89219e1bd9b8546ceacc/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3fcd603c438a02ea367739c63738ff450363e224612d0293dea19ad32669909e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/11f5b643998dc5dc2d048abf8abdb925b89cfd87a5d63a95dc99898bc813cf24/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/54e80358c5884c86e987426bf1ffc9b60bc4af9f7c7045b6112c69fa8ed92373/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/280ebfe8d11ab28e7d13f5aa0c37bcaaf140e92299dadaf5d3264a3bf9a36090/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7854d8c11c0da7ebeef7a5110f501d5937fd499ab84de5349b955796046358fc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/1715554d511360b295a0546aac038e3bd6752d8bc6e4514bfa0638297544756b/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/38cb75d2d4b96ae08d56a987156576067fffb3910a62cd8e4d2f6181604d9e1e/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e004fd04aa85bf63f2ba6b2b81559f70db6a646d698182fb1fb2ae1eb6a85a1a/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d3323682dd5b7b9d29df655e847387f154908ecc314be7fd26c29bf0cc523256/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b4e1a933a6204f08a293cad1d8a5c2f76ee26b612e1c7abdb87a1118c1375e93/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/d29cae62c13d1d68d699be4bf54e236aa0f168b4d81cd3cdd4816abdcf97210c/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/cea9fef194e01939c98d9f7d137a7e2b4040a10387b59a3b0a4c55d007ca5e02/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/8b0a7bbc383b4a16a3e6eea63eb6703b2d41a0e207fea779767bd0d04d556d3a/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/3cc112509962423a0fec13536b521037f16dd8836b0db04b44704c1179ac53e1/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/885a43aa1cd8891c9ea53ec6b9fffe78bb10491717e5db8209b7ddc558bbdcfd/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/700d66ca94585619dbae10c87bfbf4d1e8f36f31d684de7cbd12dcc4fc8876b0/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/67c3fc52721909fe446fce60e091152403a8fe333cb61f2e9fc21eb9bc90f4da/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/eb2acee25d0dba2c43b6718a6ac0637dd76e862443f639bfa3250aca67839b3f/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/d337c6f1ffdc7137e9dd0bafe44d79e60d74480a87ab1b8dce05b2a74893a466/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/62759a9d9a0e0109ea9279e1a16ff8f44a9bbcb0fb4c358e58ae978cd3210ae7/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/36f12689b4e73584982989a90ea6e3e517f1fcca44c107aa9a8ee202554019ec/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/0748ec72a61829b64723b5f8982035f125978ff5ff5fa4b225ef77d6046b82d8/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/02eac47fdce4f4aac822cfb427c28b192bf31bb1c1719e1c6772e53a9d217f4c/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/4087af099970075e362632c3890cc87e89c245bf3a27340ce161eab7711824a0/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/427986bb28a03c70cab8a2d1c0637acf3e584d155446a5dae54ae1430f925dd5/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/253ef7a795b4007272bf70b3880706363c5234a54468070370f84dc05e03086e/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/7379b610ceb0310c3538772df1f6a98d076a75a06d7a2454d5672dcc7648df11/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/a6144b50b31d3b0c434099db3334ab4bd7b7e2a6b20914e0ae35e92dfcf0e31a/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/a3f0ccd9bd9182e343742408574f73ad76cd253d50e3c21436462d2c2ef45982/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/b5eef0bd9e0d661e617054891b25b892724752dcf7ec7ad148cca7bedb48565a/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/112cb09e378f963d2d297b7c0fd854a8d475bffaad9dab32715606d124b5258f/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/a8e182ccb1393305105f250cf120c39543ab5b3df2c006055860d7811b598d39/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/028daa64cd8142cde17a6b2914a1374af4926070c7368771a2620660e5632aae/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/a88579ae411db97ece6c0308efe9f251b2e446856dbc2c881b1373f52af2040c/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/2389a3b552bd7d5bcd84b6c44a57dcb4bab81c242faf8597d1a63848766273a8/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/01f60281cf199288cf9dcce4995c0755687f2ecb7e90c3bbe5630c8d480319c6/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/a790493998e0fced0905d78245e919590717e33c28aaf96b4c242dc6d405127a/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/9a211cfb27a0e9bfdc92b04d2a2224b98e3f0232e4b12885a28520ff31294ef0/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/bc9d9683035a369bd23230c4bf46979a51a0ae3d6ee0f9e2e24af44b9f71014e/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/fc403dcdd527edc27ed999349fc0e0a96b43dc52d479888d444e08491ba04423/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/257b27f9a3a342d4b1f406f34aeeaa2c031b7c4b615b5f8d2e17cda9abe8f6ab/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/5a51e62d0bec9fe666b53b182f80f05b7de75383d5deca0d388d8d3d726d4313/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/92fad3a9731426eb8f75e79cf64b3c2f7f25ea7c8eb5cfc96e3eb7147e68594b/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/dea8fa78745d69351b564e30b567419d6fdc3515550da4af3f5c8640352d9d99/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/720554439c0c5fa9198fff1bb43ea543e5493ca80df3e3d08600c3129aa31292/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/cc6eb07727952a9c4c011f47d5b28e7b6d76ebdd1f6d813e9f3305d049ea0942/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/1187f617163c1d99dfe92a622a5bf3c8f173a32153760fe1e2f071b848ebe550/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/d5d12835ca3f5f9bf797de2431a02f72bccc8dc9eeae4ffd5e0d3b81ef7960c2/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/ca68473057cc25d1372a79bb580c183865becf666746a30e5e6465c1c90c5e87/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/ee831edc534c1b9b67d9300bd9312589841584ace6e5ff06995b2624bb3612bd/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/b5fee6b013128eee590577a101a48780e59013fd068b2fd3ef627eae79985013/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-479:{mountpoint:/var/lib/containers/storage/overlay/735805941df064e23a1d5ae88affa568736218be0277c5dcedb1bb864e9e6495/merged major:0 minor:479 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/dd0cb11006cb08d57aadde79c052d466b9b35c72fda2a3d2bb0282d659415dee/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/e898547477da28c4eb094eb11090c05110c7d6c7c2733b78755569e310156122/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/2684c561b5b3368167994ab3bebb537c91cc4c20049274454bf584e61c029653/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-492:{mountpoint:/var/lib/containers/storage/overlay/921db98edfb5da080c86dd4efc998869d6c99d6ce06362f768924372e1441047/merged major:0 minor:492 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/4c5d3f79f228b9efcc2c0ed9354352c443d8dfd35f9b851302418583bcbddce0/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/fa81707b94598a8891968646e09be74aef60fc066a4c5fafd6f5ed136c42f2b5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/29f7ab66a8b3aff3925e0a67da56c6b73d28bec0cb9ba318c06f2793ac01e9b8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2f2f8cd17d57070c3e54cb9d05595cc3c16055c14d013b42d2a41e8c02e08028/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/1712cf0682a2582a33fed729e883bc01571617db906f6f00ff2862af71dd89aa/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/85acd1f1bec8114a6830de572053130621abae21f1326dac993a413d7de2c201/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/ac21d5f425c738f6fe08438bd04df8accc848a220e83911cd90dafc94a5796d6/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/743ccac6408cea4adf1316deab621bc1ce16b64464713886a7af1ef05555a564/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/a5c73c6072c6c8712d6a143583085c85d21c3b5a98c3627acea26f609e8889c2/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0fe0c5afb4391b85cc3f71a2853c932a7849db37c820367258fd5e9ab1dbc54f/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 13:49:30.264448 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.263753 5824 manager.go:217] Machine: {Timestamp:2026-10-30 13:49:30.263268338 +0000 UTC m=+0.063773774 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099814 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c2f910603bf86ed635fc1f92fcb3 SystemUUID:ec28c2f9-1060-3bf8-6ed6-35fc1f92fcb3 BootID:98b49e1e-8273-4d80-9a06-931cfc104110 Filesystems:[{Device:/var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c80b9b41-b573-4246-b0ca-fd5e7297e6f0/volumes/kubernetes.io~projected/kube-api-access-dsvz6 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~projected/kube-api-access-tm9fg DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b6eba9bacbad529c4f410fd1ab650a33136ce0427059789abac5c9d646e7d83/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~projected/kube-api-access-6xhpr DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7c7ddc7cc8d6e49ad41f1cf1bdd4d198d7a8c0529d22b1f83bfc28cd30e9a16/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volumes/kubernetes.io~projected/kube-api-access-4gjnm DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9f9b8de-895f-4e27-b089-0f97cfefdff8/volumes/kubernetes.io~projected/kube-api-access-cmqsp DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~projected/kube-api-access-xdvqn DeviceMajor:0 DeviceMinor:433 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f9fe1318118b023d5bb827c00dbafd7ec783d1458e843252002d3bebb0eb48d/userdata/shm DeviceMajor:0 DeviceMinor:441 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/461354a6ffb262a94623241533d908493ffd3c85de2a91399503f640414b7a49/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6ddd1f9a5297f1b2b31cabbe78e24dcc066e3311f6b515783eea7abcd28170b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a40f373a-ac17-46aa-affb-41fc629af3da/volumes/kubernetes.io~projected/kube-api-access-ft2d8 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:418 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:426 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:426 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/171b1fcd87d07a1dd74252030a4a51cb0af23e11ba6bdd5b097fd6fd086c634a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~projected/kube-api-access-9mshk DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b00acad6d839a339f62aeb53821ac3b0120cfb18d6c81768f866dbb2100a9eeb/userdata/shm DeviceMajor:0 DeviceMinor:445 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~projected/kube-api-access-qw7ft DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:332 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:427 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:426 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d17352c9210a8082ed7e9dd9a94e7182c5e7029b1d35ac8033c928976e31b49d/userdata/shm DeviceMajor:0 DeviceMinor:434 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/585e43fbe5f67219d4bb3d45bd9564d3dc8a604f4e21318542aabdbd110744b5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a55fa4712649caf1ac5b901febddbac3b15244d680f35749e25e8405edfbefea/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:431 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/45341aa5-a0c9-4241-ae17-fa2f1b223f74/volumes/kubernetes.io~projected/kube-api-access-xtp2p DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:426 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:425 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8e3c753d2c66acbeeb30d790d7b514dd5ddc05651808bbdfad823c0cf77f1df/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c116049c5f0051c4c255c94f95aebaa27a033149b9dbe649dcd99b2b3370f01/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e0458ec69773388dca1845aea143308502241a12b9c8d0981f677f9e946638b/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:429 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7aac1d6b69598fa39e2a64d5614ca1c7e2f159028be83206d735501d5477508/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afb9afd21cc05977b91a894bf16e7f1c413489d3ec7e2a24deb753a3396e0e35/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c80b9b41-b573-4246-b0ca-fd5e7297e6f0/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c082c71c-c2af-4b9d-bde3-35e11d42fb51/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a187132a53451c593de5a474312eb47b6f79e64d2d0fd04e87c0c58567261b9/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5cdb85a7-7e6c-4e60-bbac-dbf902150c60/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5940df8af7d84bf7c51ed70150627b5a0da2fa549fd13318dcbb2e7c63377875/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d413c8fe-be32-46db-9326-5cfb42fc987b/volumes/kubernetes.io~projected/kube-api-access-gcwsv DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b37398e-d79b-4ced-a7b6-07409e2dbdcc/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:439 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f11ccc81-ad15-4901-afe8-dbc49587fb90/volumes/kubernetes.io~projected/kube-api-access-9xhn2 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5121026b463798d4c27877cdf06084756d832e3ac9e6032893c057e9b93652e3/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f87e904-4c9c-4691-8858-dc3dfbe443af/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3df94a1d75c19881b6612fb5902ea860d3574702a8f4d289f07f476189b2c655/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f30df3c3-736e-4aad-bd2f-44a8866b3f93/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9f9b8de-895f-4e27-b089-0f97cfefdff8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:423 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72a2c31f-6357-4b42-be58-9672ec8dcecb/volumes/kubernetes.io~projected/kube-api-access-dtsfg DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~projected/kube-api-access-jj4s5 DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c59f3642-4311-4da7-b3e2-66c782e2b83b/volumes/kubernetes.io~projected/kube-api-access-ggnrl DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f75fa3f-6f80-4ea7-a2a1-205839a11f12/volumes/kubernetes.io~projected/kube-api-access-p82w4 DeviceMajor:0 DeviceMinor:440 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e08d19cc1d810131e0b376a84681647f90730ea81e2386a07e1e42538441af5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~projected/kube-api-access-szdqr DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9a3dce8249250a3aa57dcd5d6783b96305444d330df8a2c700b8f057a0e8041/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:432 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67379085-0be6-451f-b409-4e47dd190877/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:438 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a53bb1dd-634e-4c2f-9e71-306c2136c837/volumes/kubernetes.io~projected/kube-api-access-4h4wd DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:430 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e9025a0-9daa-426a-b18d-3a04a154b693/volumes/kubernetes.io~projected/kube-api-access-qjs8f DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67379085-0be6-451f-b409-4e47dd190877/volumes/kubernetes.io~projected/kube-api-access-dpq4l DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc933f491eac7b20ed2e7a31879bdc18af87108dda08a83857cf02534067f187/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ff2a8b95f6634d328e971d1c6be6484e6c41c18d3e2fc269be006d38c54a004/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d35bccc-a939-4913-8903-63b51885cc9a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:346 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35/userdata/shm DeviceMajor:0 DeviceMinor:313 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:428 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-479 DeviceMajor:0 DeviceMinor:479 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-492 DeviceMajor:0 DeviceMinor:492 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c082c71c-c2af-4b9d-bde3-35e11d42fb51/volumes/kubernetes.io~projected/kube-api-access-s5qvh DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80844f10-0b8f-4f5e-bbba-4a93bb0a8626/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:424 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3df94a1d75c1988 MacAddress:ca:2a:a3:94:98:19 Speed:10000 Mtu:8901} {Name:4ff2a8b95f6634d MacAddress:52:96:75:cc:7e:ba Speed:10000 Mtu:8901} {Name:5121026b463798d MacAddress:46:b2:b6:3f:70:88 Speed:10000 Mtu:8901} {Name:5f9fe1318118b02 MacAddress:a2:60:06:0f:11:c6 Speed:10000 Mtu:8901} {Name:8b6eba9bacbad52 MacAddress:46:22:8d:89:be:38 Speed:10000 Mtu:8901} {Name:9a187132a53451c MacAddress:4e:3e:85:e8:d6:79 Speed:10000 Mtu:8901} {Name:9e0458ec6977338 MacAddress:e6:3b:ec:60:eb:eb Speed:10000 Mtu:8901} {Name:a7c7ddc7cc8d6e4 MacAddress:da:57:c6:e8:09:5d Speed:10000 Mtu:8901} {Name:a9a3dce8249250a MacAddress:ce:e0:4f:2a:aa:9c Speed:10000 Mtu:8901} {Name:b00acad6d839a33 MacAddress:1a:7c:35:1a:54:5b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c7:4a:95:ff Speed:0 Mtu:9001} {Name:br-int MacAddress:82:bc:e9:da:fe:cd Speed:0 Mtu:8901} {Name:c01f5eb654e0041 MacAddress:a6:78:87:57:63:4b Speed:10000 Mtu:8901} {Name:d17352c9210a808 MacAddress:e2:be:27:40:ea:d0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c7:4a:95:ff Speed:0 Mtu:9001} {Name:f6ddd1f9a5297f1 MacAddress:12:9f:68:9d:98:10 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:22:c5:29:f1:a3:eb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e6:17:f7:19:b9:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 13:49:30.264448 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264268 5824 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 13:49:30.264448 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264325 5824 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 13:49:30.264761 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264642 5824 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 13:49:30.264816 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264677 5824 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 13:49:30.264857 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264817 5824 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 13:49:30.264857 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264825 5824 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 13:49:30.264857 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264839 5824 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:49:30.264857 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264852 5824 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 13:49:30.264974 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.264965 5824 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 13:49:30.265139 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.265129 5824 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 13:49:30.265293 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.265283 5824 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 13:49:30.265328 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.265322 5824 kubelet.go:493] "Attempting to sync node with API server" Oct 30 13:49:30.265362 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.265336 5824 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 13:49:30.265362 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.265347 5824 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 13:49:30.265362 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.265357 5824 kubelet.go:405] "Adding apiserver pod source" Oct 30 13:49:30.265461 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.265366 5824 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 13:49:30.266034 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.266018 5824 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 13:49:30.266034 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.266034 5824 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:49:30.266684 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.266647 5824 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 13:49:30.269523 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.269504 5824 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 13:49:30.269523 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.269520 5824 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 13:49:30.269683 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.269643 5824 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 13:49:30.269901 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.269883 5824 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 13:49:30.269965 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.269925 5824 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 13:49:30.270062 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270050 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 13:49:30.270116 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270070 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 13:49:30.270116 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270080 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 13:49:30.270116 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270090 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 13:49:30.270116 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270100 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 13:49:30.270116 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270110 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 13:49:30.270373 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270121 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 13:49:30.270373 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270131 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 13:49:30.270373 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270143 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 13:49:30.270373 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270153 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 13:49:30.270373 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270177 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 13:49:30.270373 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270216 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 13:49:30.270373 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270255 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 13:49:30.270373 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.270266 5824 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 13:49:30.274035 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.274017 5824 server.go:1263] "Started kubelet" Oct 30 13:49:30.274136 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.274085 5824 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 13:49:30.274691 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.274612 5824 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 13:49:30.274680 ip-10-0-131-200 systemd[1]: Started Kubernetes Kubelet. Oct 30 13:49:30.274926 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.274741 5824 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 13:49:30.275652 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.275237 5824 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 13:49:30.277121 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.277041 5824 server.go:317] "Adding debug handlers to kubelet server" Oct 30 13:49:30.280404 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.280385 5824 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 13:49:30.280495 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.280418 5824 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 13:49:30.280495 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.280481 5824 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 13:43:21 +0000 UTC" deadline="2028-04-17 06:49:42.172774403 +0000 UTC" Oct 30 13:49:30.280588 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.280504 5824 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12833h0m11.892274449s" Oct 30 13:49:30.280588 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.280543 5824 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 13:49:30.280588 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:30.280543 5824 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-200.ec2.internal\" not found" Oct 30 13:49:30.280588 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.280551 5824 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 13:49:30.280757 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.280563 5824 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 13:49:30.282903 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.282877 5824 factory.go:55] Registering systemd factory Oct 30 13:49:30.283000 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.282910 5824 factory.go:223] Registration of the systemd container factory successfully Oct 30 13:49:30.284269 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284230 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" volumeName="kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config" seLinuxMountContext="" Oct 30 13:49:30.284336 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284273 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" volumeName="kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca" seLinuxMountContext="" Oct 30 13:49:30.284336 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284286 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 13:49:30.284336 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284299 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config" seLinuxMountContext="" Oct 30 13:49:30.284336 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284315 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a40f373a-ac17-46aa-affb-41fc629af3da" volumeName="kubernetes.io/configmap/a40f373a-ac17-46aa-affb-41fc629af3da-iptables-alerter-script" seLinuxMountContext="" Oct 30 13:49:30.284336 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284330 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53bb1dd-634e-4c2f-9e71-306c2136c837" volumeName="kubernetes.io/projected/a53bb1dd-634e-4c2f-9e71-306c2136c837-kube-api-access-4h4wd" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284345 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d413c8fe-be32-46db-9326-5cfb42fc987b" volumeName="kubernetes.io/projected/d413c8fe-be32-46db-9326-5cfb42fc987b-kube-api-access-gcwsv" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284357 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d413c8fe-be32-46db-9326-5cfb42fc987b" volumeName="kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-tls" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284373 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284386 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284400 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/empty-dir/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-tmp" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284412 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c082c71c-c2af-4b9d-bde3-35e11d42fb51" volumeName="kubernetes.io/configmap/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284424 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284437 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" volumeName="kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284453 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" volumeName="kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284468 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f30df3c3-736e-4aad-bd2f-44a8866b3f93" volumeName="kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-config" seLinuxMountContext="" Oct 30 13:49:30.284487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284483 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f87e904-4c9c-4691-8858-dc3dfbe443af" volumeName="kubernetes.io/projected/0f87e904-4c9c-4691-8858-dc3dfbe443af-kube-api-access-9mshk" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284496 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284511 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45341aa5-a0c9-4241-ae17-fa2f1b223f74" volumeName="kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-tls" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284525 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d35bccc-a939-4913-8903-63b51885cc9a" volumeName="kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284537 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a40f373a-ac17-46aa-affb-41fc629af3da" volumeName="kubernetes.io/projected/a40f373a-ac17-46aa-affb-41fc629af3da-kube-api-access-ft2d8" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284627 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c59f3642-4311-4da7-b3e2-66c782e2b83b" volumeName="kubernetes.io/projected/c59f3642-4311-4da7-b3e2-66c782e2b83b-kube-api-access-ggnrl" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284645 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d413c8fe-be32-46db-9326-5cfb42fc987b" volumeName="kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284659 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" volumeName="kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284674 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80844f10-0b8f-4f5e-bbba-4a93bb0a8626" volumeName="kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284690 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c082c71c-c2af-4b9d-bde3-35e11d42fb51" volumeName="kubernetes.io/empty-dir/c082c71c-c2af-4b9d-bde3-35e11d42fb51-data-volume" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284703 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284715 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" volumeName="kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284737 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9f9b8de-895f-4e27-b089-0f97cfefdff8" volumeName="kubernetes.io/secret/e9f9b8de-895f-4e27-b089-0f97cfefdff8-cert" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284768 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11ccc81-ad15-4901-afe8-dbc49587fb90" volumeName="kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-cni-binary-copy" seLinuxMountContext="" Oct 30 13:49:30.284792 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284781 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11ccc81-ad15-4901-afe8-dbc49587fb90" volumeName="kubernetes.io/projected/f11ccc81-ad15-4901-afe8-dbc49587fb90-kube-api-access-9xhn2" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284793 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f30df3c3-736e-4aad-bd2f-44a8866b3f93" volumeName="kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-script-lib" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284819 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284831 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284844 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284857 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cdb85a7-7e6c-4e60-bbac-dbf902150c60" volumeName="kubernetes.io/configmap/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-konnectivity-ca" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284867 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284880 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284891 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b37398e-d79b-4ced-a7b6-07409e2dbdcc" volumeName="kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284902 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cdb85a7-7e6c-4e60-bbac-dbf902150c60" volumeName="kubernetes.io/secret/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-agent-certs" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284914 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284925 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c80b9b41-b573-4246-b0ca-fd5e7297e6f0" volumeName="kubernetes.io/configmap/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-config-volume" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284936 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284949 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f30df3c3-736e-4aad-bd2f-44a8866b3f93" volumeName="kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-env-overrides" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284961 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f30df3c3-736e-4aad-bd2f-44a8866b3f93" volumeName="kubernetes.io/projected/f30df3c3-736e-4aad-bd2f-44a8866b3f93-kube-api-access-4gjnm" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284973 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284986 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.284998 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45341aa5-a0c9-4241-ae17-fa2f1b223f74" volumeName="kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285010 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67379085-0be6-451f-b409-4e47dd190877" volumeName="kubernetes.io/projected/67379085-0be6-451f-b409-4e47dd190877-kube-api-access-dpq4l" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285022 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80844f10-0b8f-4f5e-bbba-4a93bb0a8626" volumeName="kubernetes.io/empty-dir/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-volume-directive-shadow" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285034 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285045 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d35bccc-a939-4913-8903-63b51885cc9a" volumeName="kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285058 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c082c71c-c2af-4b9d-bde3-35e11d42fb51" volumeName="kubernetes.io/secret/c082c71c-c2af-4b9d-bde3-35e11d42fb51-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285070 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c59f3642-4311-4da7-b3e2-66c782e2b83b" volumeName="kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-binary-copy" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285082 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285094 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45341aa5-a0c9-4241-ae17-fa2f1b223f74" volumeName="kubernetes.io/configmap/45341aa5-a0c9-4241-ae17-fa2f1b223f74-metrics-client-ca" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285116 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/configmap/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-metrics-client-ca" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285128 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285141 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d35bccc-a939-4913-8903-63b51885cc9a" volumeName="kubernetes.io/projected/9d35bccc-a939-4913-8903-63b51885cc9a-kube-api-access-6xhpr" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285152 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c082c71c-c2af-4b9d-bde3-35e11d42fb51" volumeName="kubernetes.io/projected/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-api-access-s5qvh" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285165 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c80b9b41-b573-4246-b0ca-fd5e7297e6f0" volumeName="kubernetes.io/projected/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-kube-api-access-dsvz6" seLinuxMountContext="" Oct 30 13:49:30.285175 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285177 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285211 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45341aa5-a0c9-4241-ae17-fa2f1b223f74" volumeName="kubernetes.io/projected/45341aa5-a0c9-4241-ae17-fa2f1b223f74-kube-api-access-xtp2p" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285224 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c59f3642-4311-4da7-b3e2-66c782e2b83b" volumeName="kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285236 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d413c8fe-be32-46db-9326-5cfb42fc987b" volumeName="kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-metrics-client-ca" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285247 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d413c8fe-be32-46db-9326-5cfb42fc987b" volumeName="kubernetes.io/empty-dir/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-textfile" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285259 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285270 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f87e904-4c9c-4691-8858-dc3dfbe443af" volumeName="kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-tuned" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285281 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285293 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72a2c31f-6357-4b42-be58-9672ec8dcecb" volumeName="kubernetes.io/empty-dir/72a2c31f-6357-4b42-be58-9672ec8dcecb-tmp-dir" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285306 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e9025a0-9daa-426a-b18d-3a04a154b693" volumeName="kubernetes.io/projected/7e9025a0-9daa-426a-b18d-3a04a154b693-kube-api-access-qjs8f" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285317 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285336 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285347 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9f9b8de-895f-4e27-b089-0f97cfefdff8" volumeName="kubernetes.io/projected/e9f9b8de-895f-4e27-b089-0f97cfefdff8-kube-api-access-cmqsp" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285358 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11ccc81-ad15-4901-afe8-dbc49587fb90" volumeName="kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-daemon-config" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285369 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72a2c31f-6357-4b42-be58-9672ec8dcecb" volumeName="kubernetes.io/projected/72a2c31f-6357-4b42-be58-9672ec8dcecb-kube-api-access-dtsfg" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285380 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d35bccc-a939-4913-8903-63b51885cc9a" volumeName="kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285391 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d413c8fe-be32-46db-9326-5cfb42fc987b" volumeName="kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285403 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" volumeName="kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285414 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f30df3c3-736e-4aad-bd2f-44a8866b3f93" volumeName="kubernetes.io/secret/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285426 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80844f10-0b8f-4f5e-bbba-4a93bb0a8626" volumeName="kubernetes.io/projected/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-api-access-qw7ft" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285437 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80844f10-0b8f-4f5e-bbba-4a93bb0a8626" volumeName="kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285451 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d35bccc-a939-4913-8903-63b51885cc9a" volumeName="kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285462 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285474 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f75fa3f-6f80-4ea7-a2a1-205839a11f12" volumeName="kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285493 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285506 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80844f10-0b8f-4f5e-bbba-4a93bb0a8626" volumeName="kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285517 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/projected/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-kube-api-access-jj4s5" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285533 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d35bccc-a939-4913-8903-63b51885cc9a" volumeName="kubernetes.io/empty-dir/9d35bccc-a939-4913-8903-63b51885cc9a-audit-log" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285544 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285556 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285568 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285586 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285599 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67379085-0be6-451f-b409-4e47dd190877" volumeName="kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285611 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80844f10-0b8f-4f5e-bbba-4a93bb0a8626" volumeName="kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-metrics-client-ca" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285623 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c59f3642-4311-4da7-b3e2-66c782e2b83b" volumeName="kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285635 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f87e904-4c9c-4691-8858-dc3dfbe443af" volumeName="kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-tmp" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285647 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285658 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285669 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285681 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99134c6b-8fa1-443d-ac56-a69a2bfcc9cb" volumeName="kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-grpc-tls" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285694 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d35bccc-a939-4913-8903-63b51885cc9a" volumeName="kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-client-ca-bundle" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285706 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c80b9b41-b573-4246-b0ca-fd5e7297e6f0" volumeName="kubernetes.io/secret/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-metrics-tls" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285719 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285730 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285742 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242e5d46-f768-4a1e-b7ce-2d5f56690421" volumeName="kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285754 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e9025a0-9daa-426a-b18d-3a04a154b693" volumeName="kubernetes.io/configmap/7e9025a0-9daa-426a-b18d-3a04a154b693-serviceca" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285766 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c80b9b41-b573-4246-b0ca-fd5e7297e6f0" volumeName="kubernetes.io/empty-dir/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-tmp-dir" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285778 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285791 5824 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc254d0c-5015-4683-98fa-c423c989b20c" volumeName="kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285805 5824 reconstruct.go:97] "Volume reconstruction finished" Oct 30 13:49:30.286004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.285813 5824 reconciler.go:29] "Reconciler: start to sync state" Oct 30 13:49:30.287654 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.286388 5824 factory.go:153] Registering CRI-O factory Oct 30 13:49:30.287654 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.286404 5824 factory.go:223] Registration of the crio container factory successfully Oct 30 13:49:30.287654 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.286458 5824 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 13:49:30.287654 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.286478 5824 factory.go:103] Registering Raw factory Oct 30 13:49:30.287654 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.286493 5824 manager.go:1199] Started watching for new ooms in manager Oct 30 13:49:30.287654 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.286808 5824 manager.go:319] Starting recovery of all containers Oct 30 13:49:30.307254 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.307177 5824 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 13:49:30.308640 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.308609 5824 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 13:49:30.308640 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.308643 5824 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 13:49:30.308824 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.308666 5824 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 13:49:30.308824 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:30.308710 5824 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 13:49:30.333307 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.333276 5824 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:49:30.333528 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.333509 5824 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 13:49:30.334147 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.334108 5824 generic.go:356] "Generic (PLEG): container finished" podID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerID="2c8f7899ec329fdbb67f00de00c7ca26809a829224a26248a51dbf69efaf3b46" exitCode=0 Oct 30 13:49:30.334526 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.334509 5824 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:49:30.337832 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.337799 5824 generic.go:356] "Generic (PLEG): container finished" podID="dc254d0c-5015-4683-98fa-c423c989b20c" containerID="3f03b57551d6d4f6f439ad4eac95479e996327e2b3a62befbb59e16ec6aa7551" exitCode=0 Oct 30 13:49:30.348176 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.348147 5824 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:49:30.351963 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.351921 5824 generic.go:356] "Generic (PLEG): container finished" podID="d413c8fe-be32-46db-9326-5cfb42fc987b" containerID="383796b376655add5e6ec7133964db66b5c2a332319d03cf43c40daf215466d6" exitCode=0 Oct 30 13:49:30.359366 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.359341 5824 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 13:49:30.374242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.374205 5824 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="4bca3408976837265c19d7673a09253998a6c0992873052d4445ad1489357b2c" exitCode=0 Oct 30 13:49:30.374242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.374237 5824 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="e60d1665e8d9e5b3d5c9139ecfca33a123384b88cb5f0690ca828cf9529090a7" exitCode=0 Oct 30 13:49:30.374242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.374244 5824 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="b06d25e3dd0c44e8ac4d999437a27c2ea33cba0b4ccdb8ed56584de637cefc5c" exitCode=0 Oct 30 13:49:30.374242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.374251 5824 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="cf9e69647da510af7dca19cd468b24d3356c6fa931d5c923c459938edc786608" exitCode=0 Oct 30 13:49:30.374518 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.374257 5824 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="05548e25457870a62d07796f9dd7063079f6a0f0216c0c09b9c37756b010c6e9" exitCode=0 Oct 30 13:49:30.374518 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.374263 5824 generic.go:356] "Generic (PLEG): container finished" podID="c59f3642-4311-4da7-b3e2-66c782e2b83b" containerID="8329261c64a72f701b66f62bb24ecd32ed91c80112501becde4a5de1d2735a01" exitCode=0 Oct 30 13:49:30.378835 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.378801 5824 generic.go:356] "Generic (PLEG): container finished" podID="950f8228a09238446a40b1b7667155e5" containerID="9e1fa68a73061b84bc63920775b485c771d8641b78dd81c972a9256fd6c42709" exitCode=0 Oct 30 13:49:30.408918 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:30.408847 5824 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 13:49:30.412364 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.412343 5824 manager.go:324] Recovery completed Oct 30 13:49:30.437638 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437611 5824 cpu_manager.go:226] "Starting" policy="none" Oct 30 13:49:30.437638 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437628 5824 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 13:49:30.437821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437646 5824 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 13:49:30.437821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437786 5824 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 13:49:30.437821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437797 5824 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 13:49:30.437821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437816 5824 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 13:49:30.437821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437821 5824 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 13:49:30.437972 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437829 5824 policy_none.go:50] "Start" Oct 30 13:49:30.437972 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437843 5824 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 13:49:30.437972 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.437855 5824 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 13:49:30.438967 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.438954 5824 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 13:49:30.439004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.438968 5824 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 13:49:30.439004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.438974 5824 policy_none.go:44] "Start" Oct 30 13:49:30.445427 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.445393 5824 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 13:49:30.450074 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.450055 5824 manager.go:341] "Starting Device Plugin manager" Oct 30 13:49:30.450571 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:30.450092 5824 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 13:49:30.450571 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.450572 5824 server.go:87] "Starting device plugin registration server" Oct 30 13:49:30.450862 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.450849 5824 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 13:49:30.450899 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.450862 5824 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 13:49:30.450980 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.450956 5824 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 13:49:30.451104 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.451071 5824 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 13:49:30.451161 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.451117 5824 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 13:49:30.451161 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.451125 5824 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 13:49:30.451288 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.451209 5824 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:49:30.451102248Z","UUID":"5885468a-188f-4101-b392-f1280be0d00b","Handler":null,"Name":"","Endpoint":""} Oct 30 13:49:30.452657 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.452641 5824 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 13:49:30.452739 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.452672 5824 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 13:49:30.452874 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:30.452852 5824 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 13:49:30.551608 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.551570 5824 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 13:49:30.553810 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.553793 5824 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientMemory" Oct 30 13:49:30.553889 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.553824 5824 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 13:49:30.553889 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.553836 5824 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-200.ec2.internal" event="NodeHasSufficientPID" Oct 30 13:49:30.553963 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.553893 5824 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.596209 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.596170 5824 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.596308 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.596257 5824 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.609432 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.609387 5824 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal"] Oct 30 13:49:30.610104 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.610070 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" event={"ID":"bda9b1f2dbe238b1af8586cc49b37f56","Type":"ContainerStarted","Data":"6d8a33d1f4c535dbcabe6eb5588b384a06ace8d04fbd03026e57e5388ce6dd98"} Oct 30 13:49:30.610168 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.610119 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" event={"ID":"bda9b1f2dbe238b1af8586cc49b37f56","Type":"ContainerStarted","Data":"171b1fcd87d07a1dd74252030a4a51cb0af23e11ba6bdd5b097fd6fd086c634a"} Oct 30 13:49:30.610168 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.610139 5824 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.610272 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.610166 5824 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.610272 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.610260 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" event={"ID":"950f8228a09238446a40b1b7667155e5","Type":"ContainerStarted","Data":"d0556c2bf234383a9080910ed69574b29dbd8fb1e72b9582292ac5c86212a04a"} Oct 30 13:49:30.610345 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.610278 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" event={"ID":"950f8228a09238446a40b1b7667155e5","Type":"ContainerDied","Data":"9e1fa68a73061b84bc63920775b485c771d8641b78dd81c972a9256fd6c42709"} Oct 30 13:49:30.610345 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.610290 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" event={"ID":"950f8228a09238446a40b1b7667155e5","Type":"ContainerStarted","Data":"b7aac1d6b69598fa39e2a64d5614ca1c7e2f159028be83206d735501d5477508"} Oct 30 13:49:30.639392 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.639368 5824 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:49:30.639484 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:30.639414 5824 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.640861 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.640847 5824 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:49:30.640902 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:30.640885 5824 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.735210 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.735172 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.735306 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.735224 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.735306 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.735252 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bda9b1f2dbe238b1af8586cc49b37f56-config\") pod \"kube-apiserver-proxy-ip-10-0-131-200.ec2.internal\" (UID: \"bda9b1f2dbe238b1af8586cc49b37f56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.835874 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.835838 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bda9b1f2dbe238b1af8586cc49b37f56-config\") pod \"kube-apiserver-proxy-ip-10-0-131-200.ec2.internal\" (UID: \"bda9b1f2dbe238b1af8586cc49b37f56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.835976 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.835919 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.835976 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.835930 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.835976 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.835943 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bda9b1f2dbe238b1af8586cc49b37f56-config\") pod \"kube-apiserver-proxy-ip-10-0-131-200.ec2.internal\" (UID: \"bda9b1f2dbe238b1af8586cc49b37f56\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.835976 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.835971 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:30.836125 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:30.836004 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950f8228a09238446a40b1b7667155e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" (UID: \"950f8228a09238446a40b1b7667155e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:31.265566 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.265532 5824 apiserver.go:52] "Watching apiserver" Oct 30 13:49:31.325092 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.325062 5824 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 13:49:31.325683 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.325663 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-stdsc","openshift-multus/multus-additional-cni-plugins-6wf4d","openshift-monitoring/alertmanager-main-0","openshift-console/console-7bcd85777b-rpw8q","openshift-dns/node-resolver-p7256","openshift-monitoring/metrics-server-76d44865b4-8hgxr","openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79","kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj","openshift-multus/multus-qb4l7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal","openshift-monitoring/node-exporter-dsrlt","openshift-image-registry/node-ca-pscjw","openshift-ingress-canary/ingress-canary-m5qfs","openshift-monitoring/prometheus-operator-9cd89b74c-4dblz","openshift-cluster-node-tuning-operator/tuned-8b7mh","openshift-network-operator/iptables-alerter-tghzm","openshift-ovn-kubernetes/ovnkube-node-2l4sz","openshift-dns/dns-default-7n7tm","openshift-network-diagnostics/network-check-target-ldf6b","kube-system/global-pull-secret-syncer-pb5tm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5","openshift-insights/insights-runtime-extractor-6ltg7","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-gwvzz"] Oct 30 13:49:31.343806 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.343780 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.343806 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.343802 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.343995 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.343847 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.343995 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.343870 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 13:49:31.343995 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.343903 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.344147 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.343994 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.344274 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.344259 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.344274 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.344266 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.344674 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.344653 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.344845 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.344694 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.344845 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.344707 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.344920 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.344874 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 13:49:31.345261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345241 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jc894\"" Oct 30 13:49:31.345261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345242 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gw59s\"" Oct 30 13:49:31.345261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345268 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smj8j\"" Oct 30 13:49:31.345413 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345275 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xn5x9\"" Oct 30 13:49:31.345413 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345316 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 13:49:31.345413 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345342 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 13:49:31.345498 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345427 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5pv6\"" Oct 30 13:49:31.345498 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345476 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w8shz\"" Oct 30 13:49:31.345564 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345518 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mxshj\"" Oct 30 13:49:31.345564 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.345548 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 13:49:31.355859 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.355844 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 13:49:31.355963 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.355950 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.356142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.356127 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7j7wh\"" Oct 30 13:49:31.356494 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.356483 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 13:49:31.357031 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.357013 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 13:49:31.357600 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.357586 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.357668 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.357620 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-697m8\"" Oct 30 13:49:31.358774 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.358758 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 13:49:31.359348 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.359334 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 13:49:31.359407 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.359367 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 13:49:31.359639 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.359619 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 13:49:31.359970 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.359956 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-58vdnq5moqt0o\"" Oct 30 13:49:31.360007 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.359978 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 13:49:31.360007 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.359988 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 13:49:31.360184 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360170 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k4gxw\"" Oct 30 13:49:31.360434 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360412 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 13:49:31.360540 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360495 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 13:49:31.360581 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360550 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 13:49:31.360581 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360563 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 13:49:31.360581 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360566 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6lwlb\"" Oct 30 13:49:31.360688 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360555 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.360688 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360639 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 13:49:31.360688 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360668 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.360688 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360659 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 13:49:31.360818 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360691 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 13:49:31.360818 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360783 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 13:49:31.360870 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360821 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w6ztc\"" Oct 30 13:49:31.360898 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360876 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8r92s\"" Oct 30 13:49:31.360946 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360935 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qrpwr\"" Oct 30 13:49:31.360976 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.360950 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a76qod3b0ca7e\"" Oct 30 13:49:31.361026 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.361016 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-f6xjc\"" Oct 30 13:49:31.361058 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.361052 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 13:49:31.362663 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.362650 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 13:49:31.362753 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.362740 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 13:49:31.362988 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.362967 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.363088 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363016 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 13:49:31.363151 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363095 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 13:49:31.363151 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363102 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 13:49:31.363274 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363181 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.363274 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363208 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 13:49:31.363274 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363238 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.363421 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363310 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5r2q6\"" Oct 30 13:49:31.363421 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363344 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 13:49:31.363421 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363408 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 13:49:31.363421 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363422 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.363603 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363485 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 13:49:31.363603 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363495 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 13:49:31.363603 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.363573 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.370871 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.370851 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 13:49:31.372555 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.372536 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 13:49:31.373402 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.373385 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.373402 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.373396 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 13:49:31.373504 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.373425 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 13:49:31.373504 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.373429 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 13:49:31.373504 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.373450 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 13:49:31.373504 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.373431 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 13:49:31.375109 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.375088 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hdbtp\"" Oct 30 13:49:31.375236 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.375221 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 13:49:31.384153 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.384134 5824 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 13:49:31.388010 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.387990 5824 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:49:31.388125 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.388112 5824 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:31.422927 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.422906 5824 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:49:31.423010 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.422925 5824 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 13:49:31.423010 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.422955 5824 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-200.ec2.internal" Oct 30 13:49:31.423010 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.422966 5824 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-200.ec2.internal" Oct 30 13:49:31.440507 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440478 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:31.440607 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440522 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-system-cni-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.440607 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440550 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082c71c-c2af-4b9d-bde3-35e11d42fb51-data-volume\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.440607 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440577 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.440746 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440605 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.440746 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440632 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-multus\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.440746 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440674 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-conf-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.440746 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440699 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmqsp\" (UniqueName: \"kubernetes.io/projected/e9f9b8de-895f-4e27-b089-0f97cfefdff8-kube-api-access-cmqsp\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:31.440746 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440724 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.440912 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440740 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c082c71c-c2af-4b9d-bde3-35e11d42fb51-data-volume\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.440912 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440748 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.440912 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440866 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.440912 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440903 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.441058 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.440982 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-registration-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.441058 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441008 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.441058 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441033 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-sys\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.441215 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441086 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.441215 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441121 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.441215 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441164 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.441350 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441224 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-metrics-client-ca\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.441350 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441231 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.441350 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441247 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e9025a0-9daa-426a-b18d-3a04a154b693-serviceca\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:49:31.441350 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441341 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.441543 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441358 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.441543 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441413 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-lib-modules\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.441543 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441436 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-tuned\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.441543 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441444 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-metrics-client-ca\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.441543 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441514 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-tuned\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.441771 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441550 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mshk\" (UniqueName: \"kubernetes.io/projected/0f87e904-4c9c-4691-8858-dc3dfbe443af-kube-api-access-9mshk\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.441771 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441548 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.441771 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441686 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.441771 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441718 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45341aa5-a0c9-4241-ae17-fa2f1b223f74-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:31.441771 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441756 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.441987 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441872 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-cnibin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.441987 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441916 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e9025a0-9daa-426a-b18d-3a04a154b693-serviceca\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:49:31.441987 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.441976 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-daemon-config\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.442089 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442005 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-sys\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.442089 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442033 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45341aa5-a0c9-4241-ae17-fa2f1b223f74-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:31.442173 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442096 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysconfig\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.442173 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442115 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-tmp\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.442173 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442130 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.442173 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442158 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-daemon-config\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.442173 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442165 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.442405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442214 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f87e904-4c9c-4691-8858-dc3dfbe443af-tmp\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.442405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442275 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.442405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442316 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-ovn\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.442405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442335 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-client-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.442405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442357 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-dbus\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:31.442405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442376 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.442405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442393 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a40f373a-ac17-46aa-affb-41fc629af3da-iptables-alerter-script\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442428 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c082c71c-c2af-4b9d-bde3-35e11d42fb51-crio-socket\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442467 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtp2p\" (UniqueName: \"kubernetes.io/projected/45341aa5-a0c9-4241-ae17-fa2f1b223f74-kube-api-access-xtp2p\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442510 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442554 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442595 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-client-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442610 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a40f373a-ac17-46aa-affb-41fc629af3da-iptables-alerter-script\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442636 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9d35bccc-a939-4913-8903-63b51885cc9a-audit-log\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442701 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.442726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442730 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9d35bccc-a939-4913-8903-63b51885cc9a-audit-log\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442760 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442771 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442821 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442843 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvqn\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442889 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-config-volume\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442902 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.442970 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f9b8de-895f-4e27-b089-0f97cfefdff8-cert\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443054 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443081 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-netd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.443113 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443086 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443140 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-script-lib\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443159 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443208 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-kubelet-config\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443284 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443293 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40f373a-ac17-46aa-affb-41fc629af3da-host-slash\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443301 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-config-volume\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443321 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443337 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-env-overrides\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443355 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsfg\" (UniqueName: \"kubernetes.io/projected/72a2c31f-6357-4b42-be58-9672ec8dcecb-kube-api-access-dtsfg\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443380 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2d8\" (UniqueName: \"kubernetes.io/projected/a40f373a-ac17-46aa-affb-41fc629af3da-kube-api-access-ft2d8\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.443408 5824 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443398 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-script-lib\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.443482 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:31.943462243 +0000 UTC m=+1.743967708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-config" not registered Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443505 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-env-overrides\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.443550 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443526 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443580 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443608 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-etc-kubernetes\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443635 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443675 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdqr\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443701 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-metrics-tls\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443726 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443728 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpq4l\" (UniqueName: \"kubernetes.io/projected/67379085-0be6-451f-b409-4e47dd190877-kube-api-access-dpq4l\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443796 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443822 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a2c31f-6357-4b42-be58-9672ec8dcecb-tmp-dir\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.443852 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-tls\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444039 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444070 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-wtmp\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444095 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e9025a0-9daa-426a-b18d-3a04a154b693-host\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444109 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444132 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444157 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-system-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444234 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-k8s-cni-cncf-io\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.444286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444263 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444349 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444443 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a2c31f-6357-4b42-be58-9672ec8dcecb-tmp-dir\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444485 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444656 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-konnectivity-ca\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444484 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-konnectivity-ca\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444686 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444697 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444814 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-netns\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444847 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444876 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444923 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444950 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-textfile\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.444979 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445004 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-tmp\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445017 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445082 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-grpc-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445108 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwsv\" (UniqueName: \"kubernetes.io/projected/d413c8fe-be32-46db-9326-5cfb42fc987b-kube-api-access-gcwsv\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.445106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445110 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-textfile\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445136 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-cnibin\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445171 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-tmp\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445163 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445250 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjs8f\" (UniqueName: \"kubernetes.io/projected/7e9025a0-9daa-426a-b18d-3a04a154b693-kube-api-access-qjs8f\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445276 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-sys-fs\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445302 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-var-lib-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445324 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-bin\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445350 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445371 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445376 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445441 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhn2\" (UniqueName: \"kubernetes.io/projected/f11ccc81-ad15-4901-afe8-dbc49587fb90-kube-api-access-9xhn2\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445466 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445489 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445513 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-conf\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445538 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445562 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445575 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445586 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445611 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.445625 5824 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445801 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c59f3642-4311-4da7-b3e2-66c782e2b83b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.445954 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.445843 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446020 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446133 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446418 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446463 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-tls\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446506 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-hostroot\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446535 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446563 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-agent-certs\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446587 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7ft\" (UniqueName: \"kubernetes.io/projected/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-api-access-qw7ft\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446613 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-socket-dir-parent\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446639 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446717 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446742 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnrl\" (UniqueName: \"kubernetes.io/projected/c59f3642-4311-4da7-b3e2-66c782e2b83b-kube-api-access-ggnrl\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446767 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-node-log\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446790 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-log-socket\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446813 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-kubelet\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446839 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446864 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446889 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qvh\" (UniqueName: \"kubernetes.io/projected/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-api-access-s5qvh\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446937 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446963 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-cni-binary-copy\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.446996 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447022 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-etc-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.447040 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447047 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447223 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-client-certs\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447250 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447251 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447276 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-metrics-client-ca\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447301 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4s5\" (UniqueName: \"kubernetes.io/projected/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-kube-api-access-jj4s5\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447326 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-tmp-dir\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447349 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447374 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-systemd-units\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447399 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c082c71c-c2af-4b9d-bde3-35e11d42fb51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447423 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447447 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-root\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.447458 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:31.947441192 +0000 UTC m=+1.747946637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447494 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447529 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-device-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447558 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-modprobe-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447585 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-kubernetes\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447610 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-systemd\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447622 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447636 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-var-lib-kubelet\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447662 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-slash\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447736 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447766 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447794 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-tls\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447795 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447833 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-multus-certs\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447859 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447872 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447887 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-config\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447912 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447945 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-metrics-server-audit-profiles\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447973 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.447999 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvz6\" (UniqueName: \"kubernetes.io/projected/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-kube-api-access-dsvz6\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448025 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448052 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448078 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-bin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448092 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.448142 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448101 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-kubelet\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448330 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9d35bccc-a939-4913-8903-63b51885cc9a-secret-metrics-server-client-certs\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448349 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gjnm\" (UniqueName: \"kubernetes.io/projected/f30df3c3-736e-4aad-bd2f-44a8866b3f93-kube-api-access-4gjnm\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448381 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhpr\" (UniqueName: \"kubernetes.io/projected/9d35bccc-a939-4913-8903-63b51885cc9a-kube-api-access-6xhpr\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448410 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448437 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448464 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448493 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-netns\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448517 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-socket-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448543 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67379085-0be6-451f-b409-4e47dd190877-metrics-certs\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448913 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448958 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c082c71c-c2af-4b9d-bde3-35e11d42fb51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.448998 5824 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.449022 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.449057 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:31.949042391 +0000 UTC m=+1.749547818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.449219 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovnkube-config\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.449282 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.449458 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-metrics-client-ca\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.449532 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-tmp-dir\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.449546 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-metrics-server-audit-profiles\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.449573 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.448544 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4wd\" (UniqueName: \"kubernetes.io/projected/a53bb1dd-634e-4c2f-9e71-306c2136c837-kube-api-access-4h4wd\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.449795 5824 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.449840 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:31.949825117 +0000 UTC m=+1.750330558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"service-ca" not registered Oct 30 13:49:31.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.449857 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-original-pull-secret\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450007 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d35bccc-a939-4913-8903-63b51885cc9a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450205 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f11ccc81-ad15-4901-afe8-dbc49587fb90-cni-binary-copy\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.450275 5824 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450309 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-run\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450311 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5cdb85a7-7e6c-4e60-bbac-dbf902150c60-agent-certs\") pod \"konnectivity-agent-stdsc\" (UID: \"5cdb85a7-7e6c-4e60-bbac-dbf902150c60\") " pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450336 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-host\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450364 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-systemd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450412 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.450575 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:31.95056064 +0000 UTC m=+1.751066085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450620 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450680 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72a2c31f-6357-4b42-be58-9672ec8dcecb-hosts-file\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450726 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.451082 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.450992 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.451696 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451105 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 13:49:31.451696 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451227 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 13:49:31.451696 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451319 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 13:49:31.451696 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451389 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.451696 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451652 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 13:49:31.451945 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451772 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 13:49:31.451945 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451815 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 13:49:31.451945 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451829 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 13:49:31.451945 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451854 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 13:49:31.451945 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451878 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 13:49:31.451945 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451887 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 13:49:31.451945 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451911 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-53r2k88csueue\"" Oct 30 13:49:31.451945 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451920 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.451988 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452040 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452049 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452045 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452072 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452083 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452107 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2dbfk\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452125 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452127 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452164 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7cls\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452210 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452240 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 13:49:31.452296 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452299 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-snt8q\"" Oct 30 13:49:31.452637 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452392 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.452637 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452435 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lkhsk\"" Oct 30 13:49:31.452637 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452482 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fvfjd\"" Oct 30 13:49:31.452637 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452518 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.452637 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.452510 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 13:49:31.453473 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.453453 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.453513 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.453469 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9f9b8de-895f-4e27-b089-0f97cfefdff8-cert\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:31.453773 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.453754 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.454933 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.454916 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-tls\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.454981 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.454919 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.454981 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.454951 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-metrics-tls\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:31.456112 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.456094 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-secret-grpc-tls\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.456203 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.456106 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:31.456312 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.456296 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.459327 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.459311 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.459398 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.459339 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/45341aa5-a0c9-4241-ae17-fa2f1b223f74-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:31.459511 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.459496 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f30df3c3-736e-4aad-bd2f-44a8866b3f93-ovn-node-metrics-cert\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.460411 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.460396 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.460457 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.460433 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.460457 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.460442 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.461568 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.461554 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.461617 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.461571 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.461653 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.461613 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.462971 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.462949 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.464914 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.464895 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmqsp\" (UniqueName: \"kubernetes.io/projected/e9f9b8de-895f-4e27-b089-0f97cfefdff8-kube-api-access-cmqsp\") pod \"ingress-canary-m5qfs\" (UID: \"e9f9b8de-895f-4e27-b089-0f97cfefdff8\") " pod="openshift-ingress-canary/ingress-canary-m5qfs" Oct 30 13:49:31.464914 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.464910 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mshk\" (UniqueName: \"kubernetes.io/projected/0f87e904-4c9c-4691-8858-dc3dfbe443af-kube-api-access-9mshk\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.465010 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.464959 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82w4\" (UniqueName: \"kubernetes.io/projected/1f75fa3f-6f80-4ea7-a2a1-205839a11f12-kube-api-access-p82w4\") pod \"network-check-target-ldf6b\" (UID: \"1f75fa3f-6f80-4ea7-a2a1-205839a11f12\") " pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:31.465206 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.465171 5824 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 13:49:31.465241 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.465213 5824 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 13:49:31.465241 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.465225 5824 projected.go:196] Error preparing data for projected volume kube-api-access-tm9fg for pod openshift-console/console-7bcd85777b-rpw8q: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:31.465294 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.465277 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:31.965264483 +0000 UTC m=+1.765769928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9fg" (UniqueName: "kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:31.466377 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.466361 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtp2p\" (UniqueName: \"kubernetes.io/projected/45341aa5-a0c9-4241-ae17-fa2f1b223f74-kube-api-access-xtp2p\") pod \"prometheus-operator-9cd89b74c-4dblz\" (UID: \"45341aa5-a0c9-4241-ae17-fa2f1b223f74\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-4dblz" Oct 30 13:49:31.466787 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.466772 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvqn\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn\") pod \"prometheus-k8s-0\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:31.476972 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.476952 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2d8\" (UniqueName: \"kubernetes.io/projected/a40f373a-ac17-46aa-affb-41fc629af3da-kube-api-access-ft2d8\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:49:31.477620 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.477600 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4wd\" (UniqueName: \"kubernetes.io/projected/a53bb1dd-634e-4c2f-9e71-306c2136c837-kube-api-access-4h4wd\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.477756 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.477735 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpq4l\" (UniqueName: \"kubernetes.io/projected/67379085-0be6-451f-b409-4e47dd190877-kube-api-access-dpq4l\") pod \"network-metrics-daemon-gwvzz\" (UID: \"67379085-0be6-451f-b409-4e47dd190877\") " pod="openshift-multus/network-metrics-daemon-gwvzz" Oct 30 13:49:31.477810 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.477792 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhn2\" (UniqueName: \"kubernetes.io/projected/f11ccc81-ad15-4901-afe8-dbc49587fb90-kube-api-access-9xhn2\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.482202 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.482175 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjs8f\" (UniqueName: \"kubernetes.io/projected/7e9025a0-9daa-426a-b18d-3a04a154b693-kube-api-access-qjs8f\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:49:31.482780 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.482760 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwsv\" (UniqueName: \"kubernetes.io/projected/d413c8fe-be32-46db-9326-5cfb42fc987b-kube-api-access-gcwsv\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.483003 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.482986 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7ft\" (UniqueName: \"kubernetes.io/projected/80844f10-0b8f-4f5e-bbba-4a93bb0a8626-kube-api-access-qw7ft\") pod \"kube-state-metrics-5dc8d7d89b-cz5xj\" (UID: \"80844f10-0b8f-4f5e-bbba-4a93bb0a8626\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cz5xj" Oct 30 13:49:31.483046 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.482994 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhpr\" (UniqueName: \"kubernetes.io/projected/9d35bccc-a939-4913-8903-63b51885cc9a-kube-api-access-6xhpr\") pod \"metrics-server-76d44865b4-8hgxr\" (UID: \"9d35bccc-a939-4913-8903-63b51885cc9a\") " pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:31.483335 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.483319 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsfg\" (UniqueName: \"kubernetes.io/projected/72a2c31f-6357-4b42-be58-9672ec8dcecb-kube-api-access-dtsfg\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:49:31.487576 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.487537 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnrl\" (UniqueName: \"kubernetes.io/projected/c59f3642-4311-4da7-b3e2-66c782e2b83b-kube-api-access-ggnrl\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.487666 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.487651 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qvh\" (UniqueName: \"kubernetes.io/projected/c082c71c-c2af-4b9d-bde3-35e11d42fb51-kube-api-access-s5qvh\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.487742 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.487675 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvz6\" (UniqueName: \"kubernetes.io/projected/c80b9b41-b573-4246-b0ca-fd5e7297e6f0-kube-api-access-dsvz6\") pod \"dns-default-7n7tm\" (UID: \"c80b9b41-b573-4246-b0ca-fd5e7297e6f0\") " pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:31.487742 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.487713 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gjnm\" (UniqueName: \"kubernetes.io/projected/f30df3c3-736e-4aad-bd2f-44a8866b3f93-kube-api-access-4gjnm\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.487829 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.487759 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4s5\" (UniqueName: \"kubernetes.io/projected/99134c6b-8fa1-443d-ac56-a69a2bfcc9cb-kube-api-access-jj4s5\") pod \"thanos-querier-6ddbd5f6b-7kc79\" (UID: \"99134c6b-8fa1-443d-ac56-a69a2bfcc9cb\") " pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:31.487829 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.487787 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdqr\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr\") pod \"alertmanager-main-0\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:31.551619 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551540 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-node-log\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.551619 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551580 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-log-socket\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.551619 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551606 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-kubelet\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.551814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551664 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-node-log\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.551814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551666 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-etc-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.551814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551686 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-log-socket\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.551814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551735 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-etc-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.551814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551740 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-kubelet\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.551814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551782 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551831 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-systemd-units\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551859 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-root\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551867 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551884 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-device-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551896 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-systemd-units\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551908 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-modprobe-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551935 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-root\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551933 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-device-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551935 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-kubernetes\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551977 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-systemd\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551986 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-kubernetes\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.551992 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-var-lib-kubelet\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552007 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-slash\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552025 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552036 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-systemd\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552039 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-modprobe-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552039 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-multus-certs\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552057 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-multus-certs\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552077 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552089 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-var-lib-kubelet\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552112 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-bin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552113 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-ovn-kubernetes\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552080 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-slash\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552141 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-kubelet\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552159 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-bin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552176 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-netns\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552218 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-socket-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552231 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-netns\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552175 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-kubelet\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552235 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-d\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552264 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-run\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552297 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-socket-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552323 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-host\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552333 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-run\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552361 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-systemd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552368 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-host\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552391 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72a2c31f-6357-4b42-be58-9672ec8dcecb-hosts-file\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552393 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-systemd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552417 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-system-cni-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552435 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-multus\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.552452 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552450 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72a2c31f-6357-4b42-be58-9672ec8dcecb-hosts-file\") pod \"node-resolver-p7256\" (UID: \"72a2c31f-6357-4b42-be58-9672ec8dcecb\") " pod="openshift-dns/node-resolver-p7256" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552466 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-conf-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552493 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-var-lib-cni-multus\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552482 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-system-cni-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552496 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-registration-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552535 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-conf-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552538 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552594 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-sys\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552603 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-registration-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552633 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552709 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-sys\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552750 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552784 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552802 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-lib-modules\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552828 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-cnibin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552842 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-sys\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552855 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysconfig\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552881 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-lib-modules\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552872 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-ovn\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552913 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-sys\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552922 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-cnibin\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552928 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-dbus\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552932 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysconfig\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552959 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c082c71c-c2af-4b9d-bde3-35e11d42fb51-crio-socket\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552984 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-dbus\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552922 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-ovn\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552987 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-netd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553003 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-netd\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553025 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-kubelet-config\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553053 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40f373a-ac17-46aa-affb-41fc629af3da-host-slash\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.552999 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c082c71c-c2af-4b9d-bde3-35e11d42fb51-crio-socket\") pod \"insights-runtime-extractor-6ltg7\" (UID: \"c082c71c-c2af-4b9d-bde3-35e11d42fb51\") " pod="openshift-insights/insights-runtime-extractor-6ltg7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553084 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b37398e-d79b-4ced-a7b6-07409e2dbdcc-kubelet-config\") pod \"global-pull-secret-syncer-pb5tm\" (UID: \"1b37398e-d79b-4ced-a7b6-07409e2dbdcc\") " pod="kube-system/global-pull-secret-syncer-pb5tm" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553112 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-etc-kubernetes\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553133 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-wtmp\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553135 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-etc-kubernetes\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553112 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40f373a-ac17-46aa-affb-41fc629af3da-host-slash\") pod \"iptables-alerter-tghzm\" (UID: \"a40f373a-ac17-46aa-affb-41fc629af3da\") " pod="openshift-network-operator/iptables-alerter-tghzm" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553149 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e9025a0-9daa-426a-b18d-3a04a154b693-host\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553219 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-system-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553246 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e9025a0-9daa-426a-b18d-3a04a154b693-host\") pod \"node-ca-pscjw\" (UID: \"7e9025a0-9daa-426a-b18d-3a04a154b693\") " pod="openshift-image-registry/node-ca-pscjw" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553241 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d413c8fe-be32-46db-9326-5cfb42fc987b-node-exporter-wtmp\") pod \"node-exporter-dsrlt\" (UID: \"d413c8fe-be32-46db-9326-5cfb42fc987b\") " pod="openshift-monitoring/node-exporter-dsrlt" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553246 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-k8s-cni-cncf-io\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553280 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-host-run-k8s-cni-cncf-io\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553303 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-netns\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553310 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-system-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553328 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553378 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-run-netns\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553417 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-cnibin\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553417 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-run-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553369 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-cnibin\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553466 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-sys-fs\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553481 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-var-lib-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553521 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-bin\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553523 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a53bb1dd-634e-4c2f-9e71-306c2136c837-sys-fs\") pod \"aws-ebs-csi-driver-node-m4qn5\" (UID: \"a53bb1dd-634e-4c2f-9e71-306c2136c837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4qn5" Oct 30 13:49:31.553527 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553547 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-conf\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553572 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553589 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-hostroot\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553598 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-var-lib-openvswitch\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553605 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f30df3c3-736e-4aad-bd2f-44a8866b3f93-host-cni-bin\") pod \"ovnkube-node-2l4sz\" (UID: \"f30df3c3-736e-4aad-bd2f-44a8866b3f93\") " pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553617 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553646 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-socket-dir-parent\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553661 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-cni-dir\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553662 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-hostroot\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553692 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c59f3642-4311-4da7-b3e2-66c782e2b83b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wf4d\" (UID: \"c59f3642-4311-4da7-b3e2-66c782e2b83b\") " pod="openshift-multus/multus-additional-cni-plugins-6wf4d" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553730 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f87e904-4c9c-4691-8858-dc3dfbe443af-etc-sysctl-conf\") pod \"tuned-8b7mh\" (UID: \"0f87e904-4c9c-4691-8858-dc3dfbe443af\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7mh" Oct 30 13:49:31.555028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.553739 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ccc81-ad15-4901-afe8-dbc49587fb90-multus-socket-dir-parent\") pod \"multus-qb4l7\" (UID: \"f11ccc81-ad15-4901-afe8-dbc49587fb90\") " pod="openshift-multus/multus-qb4l7" Oct 30 13:49:31.956323 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.956290 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.956500 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.956342 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.956500 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956423 5824 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:31.956500 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.956462 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.956651 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956506 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:32.95648687 +0000 UTC m=+2.756992319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:31.956651 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.956541 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.956651 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956554 5824 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 13:49:31.956651 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956564 5824 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:31.956651 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956604 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:32.956592048 +0000 UTC m=+2.757097498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"service-ca" not registered Oct 30 13:49:31.956651 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956631 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:32.956616594 +0000 UTC m=+2.757122030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:31.956838 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956662 5824 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:31.956838 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:31.956707 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:31.956838 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956767 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:32.956750353 +0000 UTC m=+2.757255793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:31.956838 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956773 5824 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 13:49:31.956838 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:31.956822 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:32.956804537 +0000 UTC m=+2.757309964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-config" not registered Oct 30 13:49:32.057831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:32.057804 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:32.057980 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.057963 5824 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 13:49:32.058037 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.057984 5824 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 13:49:32.058037 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.057993 5824 projected.go:196] Error preparing data for projected volume kube-api-access-tm9fg for pod openshift-console/console-7bcd85777b-rpw8q: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:32.058102 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.058052 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:33.058038471 +0000 UTC m=+2.858543893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9fg" (UniqueName: "kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:32.352881 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:32.352814 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7n7tm" Oct 30 13:49:32.968212 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:32.968165 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:32.968413 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:32.968227 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:32.968413 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:32.968326 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:32.968413 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968355 5824 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:32.968413 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:32.968368 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:32.968413 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968395 5824 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:32.968415 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968355 5824 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968435 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:34.968415174 +0000 UTC m=+4.768920619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968470 5824 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968477 5824 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968493 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:34.968465486 +0000 UTC m=+4.768970911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-config" not registered Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968533 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:34.96851557 +0000 UTC m=+4.769020995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968557 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:34.968550035 +0000 UTC m=+4.769055457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:32.968611 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:32.968599 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:34.968592081 +0000 UTC m=+4.769097503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"service-ca" not registered Oct 30 13:49:33.069508 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:33.069476 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:33.069689 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:33.069656 5824 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 13:49:33.069689 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:33.069682 5824 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 13:49:33.069812 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:33.069694 5824 projected.go:196] Error preparing data for projected volume kube-api-access-tm9fg for pod openshift-console/console-7bcd85777b-rpw8q: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:33.069812 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:33.069772 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:35.069740048 +0000 UTC m=+4.870245489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9fg" (UniqueName: "kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:33.639313 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:33.639278 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:49:34.387426 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:34.387398 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-stdsc" Oct 30 13:49:34.988601 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:34.988565 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:34.988601 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:34.988604 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:34.988670 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:34.988695 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:34.988725 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988742 5824 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988784 5824 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988806 5824 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988799 5824 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988838 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:38.988820112 +0000 UTC m=+8.789325551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988811 5824 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988856 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:38.988849512 +0000 UTC m=+8.789354933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-config" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988882 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:38.988873602 +0000 UTC m=+8.789379026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988903 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:38.988893886 +0000 UTC m=+8.789399323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"service-ca" not registered Oct 30 13:49:34.989072 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:34.988923 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:38.988913944 +0000 UTC m=+8.789419367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:35.089847 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:35.089817 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:35.090002 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:35.089984 5824 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 13:49:35.090038 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:35.090011 5824 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 13:49:35.090038 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:35.090024 5824 projected.go:196] Error preparing data for projected volume kube-api-access-tm9fg for pod openshift-console/console-7bcd85777b-rpw8q: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:35.090094 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:35.090078 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:39.090060269 +0000 UTC m=+8.890565709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9fg" (UniqueName: "kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:36.471148 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:36.471106 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6ddbd5f6b-7kc79" Oct 30 13:49:36.538325 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:36.538288 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:36.553636 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:36.553610 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:39.031942 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:39.031897 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:39.031948 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032028 5824 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032068 5824 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:39.032080 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032096 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:47.032082105 +0000 UTC m=+16.832587575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"service-ca" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032135 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:47.032119644 +0000 UTC m=+16.832625068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032159 5824 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032233 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:47.032221821 +0000 UTC m=+16.832727278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:39.032274 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:39.032307 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032372 5824 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032391 5824 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032402 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:47.03239443 +0000 UTC m=+16.832899865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:39.032448 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.032431 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:47.032419458 +0000 UTC m=+16.832924915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-config" not registered Oct 30 13:49:39.132694 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:39.132656 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:39.132853 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.132826 5824 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 13:49:39.132853 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.132848 5824 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 13:49:39.132937 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.132858 5824 projected.go:196] Error preparing data for projected volume kube-api-access-tm9fg for pod openshift-console/console-7bcd85777b-rpw8q: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:39.132937 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:39.132912 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:49:47.132898291 +0000 UTC m=+16.933403750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9fg" (UniqueName: "kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:39.592485 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:39.592457 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:41.137514 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:41.137434 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:49:41.603948 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:41.603915 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:49:41.604145 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:41.604131 5824 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 13:49:45.710022 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:45.709988 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:45.710769 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:45.710272 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="alertmanager" containerID="cri-o://9885dc8de5dbb15bdd5d950bec67a2bc7702eb3c391c03c4f8cdeda65a2cc1e0" gracePeriod=120 Oct 30 13:49:45.710769 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:45.710344 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy-metric" containerID="cri-o://e2d39a27ba075372648efc479afe802e20ff959aff3bf5479196010a065a141a" gracePeriod=120 Oct 30 13:49:45.710769 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:45.710372 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy-web" containerID="cri-o://de35cd767d5c4c5beee283e34ddc5072a27b59012cca6f940dc013bec5a4d201" gracePeriod=120 Oct 30 13:49:45.710769 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:45.710398 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy" containerID="cri-o://0d90113eda2a6523354a9be7c6a641a0c5cb2942634c809dde2782da5e3c9fca" gracePeriod=120 Oct 30 13:49:45.710769 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:45.710428 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="config-reloader" containerID="cri-o://6865aa4d04bdc0cf06400509ee22c9e087684bb2e160896add23c3a1a6255dd3" gracePeriod=120 Oct 30 13:49:45.710769 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:45.710433 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="prom-label-proxy" containerID="cri-o://16e70dc35b33f1c481914b0f5597ed058b073e7751d19b541a89db370cacd620" gracePeriod=120 Oct 30 13:49:46.446943 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.446909 5824 generic.go:356] "Generic (PLEG): container finished" podID="dc254d0c-5015-4683-98fa-c423c989b20c" containerID="16e70dc35b33f1c481914b0f5597ed058b073e7751d19b541a89db370cacd620" exitCode=0 Oct 30 13:49:46.446943 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.446936 5824 generic.go:356] "Generic (PLEG): container finished" podID="dc254d0c-5015-4683-98fa-c423c989b20c" containerID="e2d39a27ba075372648efc479afe802e20ff959aff3bf5479196010a065a141a" exitCode=0 Oct 30 13:49:46.446943 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.446944 5824 generic.go:356] "Generic (PLEG): container finished" podID="dc254d0c-5015-4683-98fa-c423c989b20c" containerID="0d90113eda2a6523354a9be7c6a641a0c5cb2942634c809dde2782da5e3c9fca" exitCode=0 Oct 30 13:49:46.446943 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.446949 5824 generic.go:356] "Generic (PLEG): container finished" podID="dc254d0c-5015-4683-98fa-c423c989b20c" containerID="6865aa4d04bdc0cf06400509ee22c9e087684bb2e160896add23c3a1a6255dd3" exitCode=0 Oct 30 13:49:46.447242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.446959 5824 generic.go:356] "Generic (PLEG): container finished" podID="dc254d0c-5015-4683-98fa-c423c989b20c" containerID="9885dc8de5dbb15bdd5d950bec67a2bc7702eb3c391c03c4f8cdeda65a2cc1e0" exitCode=0 Oct 30 13:49:46.447242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.446986 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerDied","Data":"16e70dc35b33f1c481914b0f5597ed058b073e7751d19b541a89db370cacd620"} Oct 30 13:49:46.447242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.447051 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerDied","Data":"e2d39a27ba075372648efc479afe802e20ff959aff3bf5479196010a065a141a"} Oct 30 13:49:46.447242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.447076 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerDied","Data":"0d90113eda2a6523354a9be7c6a641a0c5cb2942634c809dde2782da5e3c9fca"} Oct 30 13:49:46.447242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.447101 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerDied","Data":"6865aa4d04bdc0cf06400509ee22c9e087684bb2e160896add23c3a1a6255dd3"} Oct 30 13:49:46.447242 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:46.447122 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerDied","Data":"9885dc8de5dbb15bdd5d950bec67a2bc7702eb3c391c03c4f8cdeda65a2cc1e0"} Oct 30 13:49:47.107034 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.106950 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:47.107034 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.107005 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:47.107034 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.107025 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.107050 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107072 5824 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107134 5824 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107147 5824 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107167 5824 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107140 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:50:03.107125932 +0000 UTC m=+32.907631358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107251 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:50:03.107230819 +0000 UTC m=+32.907736257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-serving-cert" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107272 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:50:03.107260384 +0000 UTC m=+32.907765806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"service-ca" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.107340 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107363 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:50:03.107346882 +0000 UTC m=+32.907852310 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-oauth-config" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107393 5824 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 13:49:47.107513 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.107455 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:50:03.10744438 +0000 UTC m=+32.907949823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : object "openshift-console"/"console-config" not registered Oct 30 13:49:47.207867 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.207836 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"console-7bcd85777b-rpw8q\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:47.208038 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.208019 5824 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 13:49:47.208081 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.208042 5824 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 13:49:47.208081 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.208054 5824 projected.go:196] Error preparing data for projected volume kube-api-access-tm9fg for pod openshift-console/console-7bcd85777b-rpw8q: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:47.208147 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:49:47.208118 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg podName:de19c6ff-20d2-4b1b-a73d-418a7b4fabf6 nodeName:}" failed. No retries permitted until 2026-10-30 13:50:03.208104291 +0000 UTC m=+33.008609740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tm9fg" (UniqueName: "kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg") pod "console-7bcd85777b-rpw8q" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 13:49:47.459405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.459370 5824 generic.go:356] "Generic (PLEG): container finished" podID="dc254d0c-5015-4683-98fa-c423c989b20c" containerID="de35cd767d5c4c5beee283e34ddc5072a27b59012cca6f940dc013bec5a4d201" exitCode=0 Oct 30 13:49:47.459530 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.459435 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerDied","Data":"de35cd767d5c4c5beee283e34ddc5072a27b59012cca6f940dc013bec5a4d201"} Oct 30 13:49:47.459530 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.459505 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc254d0c-5015-4683-98fa-c423c989b20c","Type":"ContainerDied","Data":"c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35"} Oct 30 13:49:47.459606 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.459531 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c01f5eb654e00413bef19a81596789c074f25bd7c427a6cef1c20e188f705f35" Oct 30 13:49:47.481125 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.481095 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:47.611329 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611298 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611516 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611341 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611516 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611372 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611516 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611418 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611516 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611467 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611516 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611511 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611790 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611549 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611790 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611592 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611790 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611620 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611790 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611661 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611790 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611695 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.611790 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611693 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:47.611790 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611754 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.612138 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.611804 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr\") pod \"dc254d0c-5015-4683-98fa-c423c989b20c\" (UID: \"dc254d0c-5015-4683-98fa-c423c989b20c\") " Oct 30 13:49:47.612138 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.612075 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:47.612319 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.612294 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:47.612401 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.612309 5824 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-main-db\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.612401 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.612347 5824 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.615948 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.615899 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:47.616257 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.616234 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "kube-api-access-szdqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:47.616480 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.616460 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:47.616972 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.616951 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:47.617539 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.617507 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:47.617539 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.617522 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:47.617840 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.617817 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:47.617980 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.617958 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:47.621405 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.621374 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:47.630464 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.630429 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config" pod "dc254d0c-5015-4683-98fa-c423c989b20c" (UID: "dc254d0c-5015-4683-98fa-c423c989b20c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:47.713084 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713045 5824 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-main-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713084 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713078 5824 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-tls-assets\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713084 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713087 5824 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713098 5824 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713112 5824 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-cluster-tls-config\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713123 5824 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-web-config\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713132 5824 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713140 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc254d0c-5015-4683-98fa-c423c989b20c-metrics-client-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713149 5824 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc254d0c-5015-4683-98fa-c423c989b20c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713159 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szdqr\" (UniqueName: \"kubernetes.io/projected/dc254d0c-5015-4683-98fa-c423c989b20c-kube-api-access-szdqr\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:47.713339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:47.713167 5824 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc254d0c-5015-4683-98fa-c423c989b20c-config-out\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:48.463857 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.463827 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.543537 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.543506 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:48.557517 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.557485 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:48.586367 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586295 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:48.586610 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586597 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="config-reloader" Oct 30 13:49:48.586648 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586611 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="config-reloader" Oct 30 13:49:48.586648 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586622 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="init-config-reloader" Oct 30 13:49:48.586648 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586627 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="init-config-reloader" Oct 30 13:49:48.586648 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586642 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="alertmanager" Oct 30 13:49:48.586648 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586647 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="alertmanager" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586656 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy-web" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586661 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy-web" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586671 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy-metric" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586675 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy-metric" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586682 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586686 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586692 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="prom-label-proxy" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586696 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="prom-label-proxy" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586764 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="config-reloader" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586772 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="init-config-reloader" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586780 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="alertmanager" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586786 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy-web" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586792 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy-metric" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586796 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="kube-rbac-proxy" Oct 30 13:49:48.586831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.586801 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" containerName="prom-label-proxy" Oct 30 13:49:48.620989 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.620958 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:48.621178 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.621141 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.642220 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642174 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 13:49:48.642385 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642174 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 13:49:48.642385 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642231 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 13:49:48.642385 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642386 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 13:49:48.642633 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642267 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 13:49:48.642633 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642314 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6lwlb\"" Oct 30 13:49:48.642633 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642350 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 13:49:48.642633 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642246 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 13:49:48.642633 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.642480 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 13:49:48.656010 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.655984 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 13:49:48.722519 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722482 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-web-config\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.722519 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722525 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1406a3ec-1760-435e-b89a-48ffdc425a94-config-out\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.722764 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722558 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1406a3ec-1760-435e-b89a-48ffdc425a94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.722764 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722628 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.722764 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722670 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1406a3ec-1760-435e-b89a-48ffdc425a94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.722913 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722797 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6487\" (UniqueName: \"kubernetes.io/projected/1406a3ec-1760-435e-b89a-48ffdc425a94-kube-api-access-m6487\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.722913 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722851 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.722988 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722955 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.722988 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.722982 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1406a3ec-1760-435e-b89a-48ffdc425a94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.723078 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.723055 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.723134 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.723115 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-config-volume\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.723210 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.723146 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.723302 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.723287 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1406a3ec-1760-435e-b89a-48ffdc425a94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824222 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824165 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1406a3ec-1760-435e-b89a-48ffdc425a94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824222 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824224 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824241 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1406a3ec-1760-435e-b89a-48ffdc425a94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824263 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6487\" (UniqueName: \"kubernetes.io/projected/1406a3ec-1760-435e-b89a-48ffdc425a94-kube-api-access-m6487\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824290 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824311 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824327 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1406a3ec-1760-435e-b89a-48ffdc425a94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824392 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824436 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-config-volume\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824469 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824847 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824545 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1406a3ec-1760-435e-b89a-48ffdc425a94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824847 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824617 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-web-config\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824847 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824650 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1406a3ec-1760-435e-b89a-48ffdc425a94-config-out\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.824847 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824638 5824 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 13:49:48.824847 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.824748 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1406a3ec-1760-435e-b89a-48ffdc425a94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.825458 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.825430 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1406a3ec-1760-435e-b89a-48ffdc425a94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.826111 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.826080 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1406a3ec-1760-435e-b89a-48ffdc425a94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.827521 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.827489 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1406a3ec-1760-435e-b89a-48ffdc425a94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.827881 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.827855 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1406a3ec-1760-435e-b89a-48ffdc425a94-config-out\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.828029 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.827999 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.828179 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.828161 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.828322 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.828300 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.828446 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.828426 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-web-config\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.828693 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.828670 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.828876 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.828859 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.829387 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.829369 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1406a3ec-1760-435e-b89a-48ffdc425a94-config-volume\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:48.851047 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:48.850978 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6487\" (UniqueName: \"kubernetes.io/projected/1406a3ec-1760-435e-b89a-48ffdc425a94-kube-api-access-m6487\") pod \"alertmanager-main-0\" (UID: \"1406a3ec-1760-435e-b89a-48ffdc425a94\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:49.029461 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:49.029425 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 13:49:49.383649 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:49.383624 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 13:49:49.386087 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:49.386054 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1406a3ec_1760_435e_b89a_48ffdc425a94.slice/crio-8914eece600bf1bd4f117c508572112af0dd4791c48e71c80de1a27c22b21bb5 WatchSource:0}: Error finding container 8914eece600bf1bd4f117c508572112af0dd4791c48e71c80de1a27c22b21bb5: Status 404 returned error can't find the container with id 8914eece600bf1bd4f117c508572112af0dd4791c48e71c80de1a27c22b21bb5 Oct 30 13:49:49.469741 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:49.469711 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1406a3ec-1760-435e-b89a-48ffdc425a94","Type":"ContainerStarted","Data":"8914eece600bf1bd4f117c508572112af0dd4791c48e71c80de1a27c22b21bb5"} Oct 30 13:49:50.315547 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.315507 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc254d0c-5015-4683-98fa-c423c989b20c" path="/var/lib/kubelet/pods/dc254d0c-5015-4683-98fa-c423c989b20c/volumes" Oct 30 13:49:50.335345 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.335306 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:50.335612 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.335577 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="prometheus" containerID="cri-o://daa8f54616f2c0c1c7ec7a4b8e26be6240e9bfcb6d63b3108ce881c7bc9ad07c" gracePeriod=600 Oct 30 13:49:50.335748 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.335639 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy-thanos" containerID="cri-o://488fd8a8e6459f9a67b9c370a0d3a5563f10946ff27a79184ff32a7b70382cc7" gracePeriod=600 Oct 30 13:49:50.335748 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.335677 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="config-reloader" containerID="cri-o://854eb720a368331ae27bd6d29a445263329bf81f2d8a60fae03ad20f473b1e38" gracePeriod=600 Oct 30 13:49:50.335748 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.335635 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="thanos-sidecar" containerID="cri-o://10445525592b8f18be8424ef153e97cb243074b5992370fa8bf28a0af3d0ffe4" gracePeriod=600 Oct 30 13:49:50.335881 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.335720 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy-web" containerID="cri-o://9084cd1e0f4d6fc898f79c254acdca7060f8e50b63410cf8363f506248abdfb8" gracePeriod=600 Oct 30 13:49:50.335881 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.335645 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy" containerID="cri-o://8ca88eeb9f3a532167453bceed44eab8a331ed4fcd7eb250d673e67c6a934063" gracePeriod=600 Oct 30 13:49:50.475926 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.475888 5824 generic.go:356] "Generic (PLEG): container finished" podID="1406a3ec-1760-435e-b89a-48ffdc425a94" containerID="f3973d98e26e1609bc7376699b954b722040b22f9e9e400e14ec9c06071a5da0" exitCode=0 Oct 30 13:49:50.476381 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.475923 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1406a3ec-1760-435e-b89a-48ffdc425a94","Type":"ContainerDied","Data":"f3973d98e26e1609bc7376699b954b722040b22f9e9e400e14ec9c06071a5da0"} Oct 30 13:49:50.482921 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.482890 5824 generic.go:356] "Generic (PLEG): container finished" podID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerID="488fd8a8e6459f9a67b9c370a0d3a5563f10946ff27a79184ff32a7b70382cc7" exitCode=0 Oct 30 13:49:50.483020 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.482923 5824 generic.go:356] "Generic (PLEG): container finished" podID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerID="8ca88eeb9f3a532167453bceed44eab8a331ed4fcd7eb250d673e67c6a934063" exitCode=0 Oct 30 13:49:50.483020 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.482932 5824 generic.go:356] "Generic (PLEG): container finished" podID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerID="9084cd1e0f4d6fc898f79c254acdca7060f8e50b63410cf8363f506248abdfb8" exitCode=0 Oct 30 13:49:50.483020 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.482940 5824 generic.go:356] "Generic (PLEG): container finished" podID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerID="10445525592b8f18be8424ef153e97cb243074b5992370fa8bf28a0af3d0ffe4" exitCode=0 Oct 30 13:49:50.483020 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.482950 5824 generic.go:356] "Generic (PLEG): container finished" podID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerID="854eb720a368331ae27bd6d29a445263329bf81f2d8a60fae03ad20f473b1e38" exitCode=0 Oct 30 13:49:50.483020 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.482959 5824 generic.go:356] "Generic (PLEG): container finished" podID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerID="daa8f54616f2c0c1c7ec7a4b8e26be6240e9bfcb6d63b3108ce881c7bc9ad07c" exitCode=0 Oct 30 13:49:50.483020 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.482959 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerDied","Data":"488fd8a8e6459f9a67b9c370a0d3a5563f10946ff27a79184ff32a7b70382cc7"} Oct 30 13:49:50.483249 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.483034 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerDied","Data":"8ca88eeb9f3a532167453bceed44eab8a331ed4fcd7eb250d673e67c6a934063"} Oct 30 13:49:50.483249 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.483068 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerDied","Data":"9084cd1e0f4d6fc898f79c254acdca7060f8e50b63410cf8363f506248abdfb8"} Oct 30 13:49:50.483249 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.483091 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerDied","Data":"10445525592b8f18be8424ef153e97cb243074b5992370fa8bf28a0af3d0ffe4"} Oct 30 13:49:50.483249 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.483119 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerDied","Data":"854eb720a368331ae27bd6d29a445263329bf81f2d8a60fae03ad20f473b1e38"} Oct 30 13:49:50.483249 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.483146 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerDied","Data":"daa8f54616f2c0c1c7ec7a4b8e26be6240e9bfcb6d63b3108ce881c7bc9ad07c"} Oct 30 13:49:50.627752 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.627729 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:50.750725 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.750697 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.750876 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.750737 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.750876 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.750762 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.750876 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.750791 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751041 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.750914 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751041 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.750973 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751041 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751010 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751041 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751097 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751130 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751146 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:50.751261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751166 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751211 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:50.751261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751227 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751602 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751277 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751602 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751307 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751602 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751354 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751602 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751408 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751602 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751443 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751602 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751472 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751602 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751516 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.751602 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751545 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle\") pod \"242e5d46-f768-4a1e-b7ce-2d5f56690421\" (UID: \"242e5d46-f768-4a1e-b7ce-2d5f56690421\") " Oct 30 13:49:50.752019 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751628 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:50.752019 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.751853 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:50.752209 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.752163 5824 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.752288 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.752208 5824 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.752288 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.752226 5824 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.752288 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.752241 5824 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.752288 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.752248 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:50.752752 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.752721 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:50.753210 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.753159 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:50.753822 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.753798 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:50.754104 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.754074 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.754206 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.754164 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.754477 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.754450 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.755675 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.755646 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.755752 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.755708 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.755803 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.755781 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.756139 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.756117 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.756380 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.756348 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:50.756462 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.756442 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:50.756506 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.756485 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.757154 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.757131 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "kube-api-access-xdvqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:50.765590 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.765569 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config" pod "242e5d46-f768-4a1e-b7ce-2d5f56690421" (UID: "242e5d46-f768-4a1e-b7ce-2d5f56690421"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:50.853117 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853087 5824 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853117 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853116 5824 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-config-out\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853126 5824 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-metrics-client-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853135 5824 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-config\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853144 5824 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-web-config\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853153 5824 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-tls-assets\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853162 5824 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-grpc-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853171 5824 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853179 5824 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853212 5824 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-kube-rbac-proxy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853223 5824 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853234 5824 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/242e5d46-f768-4a1e-b7ce-2d5f56690421-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853242 5824 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-db\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853251 5824 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853260 5824 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/242e5d46-f768-4a1e-b7ce-2d5f56690421-configmap-metrics-client-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:50.853669 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:50.853283 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdvqn\" (UniqueName: \"kubernetes.io/projected/242e5d46-f768-4a1e-b7ce-2d5f56690421-kube-api-access-xdvqn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:51.489966 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.489929 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1406a3ec-1760-435e-b89a-48ffdc425a94","Type":"ContainerStarted","Data":"3691f4673d380f279052d6af6ab3c7f50cb8ef19e9dba0e10448824c7fe82cb8"} Oct 30 13:49:51.490433 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.490005 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1406a3ec-1760-435e-b89a-48ffdc425a94","Type":"ContainerStarted","Data":"a43be14305a0af71a634ffa0f7fd4908d3fca1ed732c31054be5256b9cf0155e"} Oct 30 13:49:51.490433 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.490044 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1406a3ec-1760-435e-b89a-48ffdc425a94","Type":"ContainerStarted","Data":"88d4c0145cc6be621458809174bca08fd7fb8193f6078ffc6a317e7997bef000"} Oct 30 13:49:51.490433 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.490070 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1406a3ec-1760-435e-b89a-48ffdc425a94","Type":"ContainerStarted","Data":"ee9371e77443798c3d7e759b05f1752f2d684cd4f4019f0cea32849d5e54df0b"} Oct 30 13:49:51.490433 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.490101 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1406a3ec-1760-435e-b89a-48ffdc425a94","Type":"ContainerStarted","Data":"14dd821b2eac7907deaa9cca96fc5ea6f5fca9ffd8bbfdf2755d1001c3d4209c"} Oct 30 13:49:51.490433 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.490131 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1406a3ec-1760-435e-b89a-48ffdc425a94","Type":"ContainerStarted","Data":"cd3e3389456167ee8520bef71bace654ae03a417e3ac778ef271052507ff57aa"} Oct 30 13:49:51.493107 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.493087 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"242e5d46-f768-4a1e-b7ce-2d5f56690421","Type":"ContainerDied","Data":"d17352c9210a8082ed7e9dd9a94e7182c5e7029b1d35ac8033c928976e31b49d"} Oct 30 13:49:51.493162 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.493144 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.493207 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.493147 5824 scope.go:122] "RemoveContainer" containerID="488fd8a8e6459f9a67b9c370a0d3a5563f10946ff27a79184ff32a7b70382cc7" Oct 30 13:49:51.501580 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.501564 5824 scope.go:122] "RemoveContainer" containerID="8ca88eeb9f3a532167453bceed44eab8a331ed4fcd7eb250d673e67c6a934063" Oct 30 13:49:51.508462 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.508445 5824 scope.go:122] "RemoveContainer" containerID="9084cd1e0f4d6fc898f79c254acdca7060f8e50b63410cf8363f506248abdfb8" Oct 30 13:49:51.514826 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.514809 5824 scope.go:122] "RemoveContainer" containerID="10445525592b8f18be8424ef153e97cb243074b5992370fa8bf28a0af3d0ffe4" Oct 30 13:49:51.521221 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.521205 5824 scope.go:122] "RemoveContainer" containerID="854eb720a368331ae27bd6d29a445263329bf81f2d8a60fae03ad20f473b1e38" Oct 30 13:49:51.527401 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.527386 5824 scope.go:122] "RemoveContainer" containerID="daa8f54616f2c0c1c7ec7a4b8e26be6240e9bfcb6d63b3108ce881c7bc9ad07c" Oct 30 13:49:51.534274 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.534256 5824 scope.go:122] "RemoveContainer" containerID="2c8f7899ec329fdbb67f00de00c7ca26809a829224a26248a51dbf69efaf3b46" Oct 30 13:49:51.549547 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.549501 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.542773146 podStartE2EDuration="3.549489026s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.71588ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:51.548249144 +0000 UTC m=+21.348754588" watchObservedRunningTime="2026-10-30 13:49:51.549489026 +0000 UTC m=+21.349994448" Oct 30 13:49:51.596169 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.596142 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:51.602099 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.602072 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:51.643317 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643291 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:51.643716 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643697 5824 cpu_manager.go:414] "containerMap: removing container" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy-thanos" Oct 30 13:49:51.643754 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643721 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy-thanos" Oct 30 13:49:51.643754 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643740 5824 cpu_manager.go:414] "containerMap: removing container" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="prometheus" Oct 30 13:49:51.643754 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643748 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="prometheus" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643774 5824 cpu_manager.go:414] "containerMap: removing container" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy-web" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643781 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy-web" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643788 5824 cpu_manager.go:414] "containerMap: removing container" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="config-reloader" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643794 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="config-reloader" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643805 5824 cpu_manager.go:414] "containerMap: removing container" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="thanos-sidecar" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643810 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="thanos-sidecar" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643818 5824 cpu_manager.go:414] "containerMap: removing container" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643822 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643828 5824 cpu_manager.go:414] "containerMap: removing container" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="init-config-reloader" Oct 30 13:49:51.643854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643832 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="init-config-reloader" Oct 30 13:49:51.644115 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643882 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy-thanos" Oct 30 13:49:51.644115 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643890 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="prometheus" Oct 30 13:49:51.644115 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643898 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy-web" Oct 30 13:49:51.644115 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643902 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="config-reloader" Oct 30 13:49:51.644115 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643909 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="thanos-sidecar" Oct 30 13:49:51.644115 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643914 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="kube-rbac-proxy" Oct 30 13:49:51.644115 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.643918 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" containerName="init-config-reloader" Oct 30 13:49:51.647235 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.647219 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.666611 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666587 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 13:49:51.666611 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666601 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 13:49:51.666611 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666604 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 13:49:51.666821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666630 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 13:49:51.666821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666643 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 13:49:51.666821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666668 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 13:49:51.666821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666746 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 13:49:51.666821 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666803 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 13:49:51.667009 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666875 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 13:49:51.667009 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666904 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 13:49:51.667009 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666906 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-58vdnq5moqt0o\"" Oct 30 13:49:51.667009 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.666982 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fvfjd\"" Oct 30 13:49:51.681070 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.681033 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 13:49:51.681261 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.681226 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 13:49:51.681466 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.681443 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 13:49:51.681776 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.681755 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 13:49:51.682587 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.682278 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:51.761814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.761726 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.761814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.761762 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.761814 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.761780 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.761857 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-web-config\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.761929 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.761951 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.761967 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762028 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.761985 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762222 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762040 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762222 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762132 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762222 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762170 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762333 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762220 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-config\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762333 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762246 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762333 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762286 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762333 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762306 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-config-out\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762341 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx29p\" (UniqueName: \"kubernetes.io/projected/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-kube-api-access-zx29p\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762391 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762408 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762487 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762426 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.762634 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.762491 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863379 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863339 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863379 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863385 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863582 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863419 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863582 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863444 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-config\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863582 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863459 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863582 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863485 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863582 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863513 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-config-out\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863582 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863538 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx29p\" (UniqueName: \"kubernetes.io/projected/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-kube-api-access-zx29p\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863600 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863625 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863656 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863687 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863713 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863732 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863750 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863780 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-web-config\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863817 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.863864 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863848 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.864303 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863879 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.864303 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.863905 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.864407 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.864338 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.864467 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.864418 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.864968 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.864929 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.867375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.866621 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.867375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.866635 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.867375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.866667 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-config\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.867375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.866936 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.867375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.867126 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-web-config\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.867375 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.867279 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.867757 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.867687 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.868051 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.868024 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-config-out\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.868051 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.868028 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.868397 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.868367 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.868613 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.868584 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.868764 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.868741 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.869172 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.869144 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.869259 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.869225 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.869778 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.869758 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.869909 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.869892 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.893457 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.893441 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx29p\" (UniqueName: \"kubernetes.io/projected/cac7de6a-e8ba-4f54-b53d-a3e16b607ac7-kube-api-access-zx29p\") pod \"prometheus-k8s-0\" (UID: \"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:51.964079 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:51.964035 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:52.308480 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:52.308449 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 13:49:52.309833 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:49:52.309803 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac7de6a_e8ba_4f54_b53d_a3e16b607ac7.slice/crio-73bd17a96cb45b09b3933bde09e9ce302d1602eb332a2c37c9a7f2a3af720ebf WatchSource:0}: Error finding container 73bd17a96cb45b09b3933bde09e9ce302d1602eb332a2c37c9a7f2a3af720ebf: Status 404 returned error can't find the container with id 73bd17a96cb45b09b3933bde09e9ce302d1602eb332a2c37c9a7f2a3af720ebf Oct 30 13:49:52.315311 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:52.315284 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242e5d46-f768-4a1e-b7ce-2d5f56690421" path="/var/lib/kubelet/pods/242e5d46-f768-4a1e-b7ce-2d5f56690421/volumes" Oct 30 13:49:52.499521 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:52.499488 5824 generic.go:356] "Generic (PLEG): container finished" podID="cac7de6a-e8ba-4f54-b53d-a3e16b607ac7" containerID="e83c600b93787d68e788d061e39ff9b091855afcf1a9aa70a2062577c3670dcc" exitCode=0 Oct 30 13:49:52.499930 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:52.499572 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7","Type":"ContainerDied","Data":"e83c600b93787d68e788d061e39ff9b091855afcf1a9aa70a2062577c3670dcc"} Oct 30 13:49:52.499930 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:52.499665 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7","Type":"ContainerStarted","Data":"73bd17a96cb45b09b3933bde09e9ce302d1602eb332a2c37c9a7f2a3af720ebf"} Oct 30 13:49:52.795737 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:52.795671 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76d44865b4-8hgxr" Oct 30 13:49:53.509146 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:53.509107 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7","Type":"ContainerStarted","Data":"ce85704244041e49af955156fe239730b642faefd1cc9863d375d173ad8ce338"} Oct 30 13:49:53.509646 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:53.509183 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7","Type":"ContainerStarted","Data":"0d7438aaa3bc96789b52dc7c1dbbdb5afe89331b7ea9ee5aea253b9f453e97b2"} Oct 30 13:49:53.509646 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:53.509246 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7","Type":"ContainerStarted","Data":"9adc79ebc5459c2bdfc8904ee494a3f9af468ed9e9a6f6769c6895bbdec00d23"} Oct 30 13:49:53.509646 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:53.509294 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7","Type":"ContainerStarted","Data":"185bac755c5e75449cfd76eb753f89302e54f9a2e0b62f30db9f8104f43eac14"} Oct 30 13:49:53.509646 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:53.509337 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7","Type":"ContainerStarted","Data":"8f010ae6ed39b838c1cefa3534b21f170287ce8a369cc41d71d07be7c5691795"} Oct 30 13:49:53.509646 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:53.509380 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cac7de6a-e8ba-4f54-b53d-a3e16b607ac7","Type":"ContainerStarted","Data":"6900c00a4e13584b4abdfff8cc01838f325735887135c55cfe83ae51ca6f4b9a"} Oct 30 13:49:53.583308 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:53.583261 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.565499564 podStartE2EDuration="2.583243812s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.744248ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:53.582336474 +0000 UTC m=+23.382841919" watchObservedRunningTime="2026-10-30 13:49:53.583243812 +0000 UTC m=+23.383749252" Oct 30 13:49:56.354236 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.354183 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7bcd85777b-rpw8q" podUID="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" containerName="console" containerID="cri-o://c2a51c895230dde8d9b5e0a162ff9eec9c7d7e7ae4978d8370137369dd0f949b" gracePeriod=15 Oct 30 13:49:56.522106 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.522073 5824 generic.go:356] "Generic (PLEG): container finished" podID="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" containerID="c2a51c895230dde8d9b5e0a162ff9eec9c7d7e7ae4978d8370137369dd0f949b" exitCode=0 Oct 30 13:49:56.522303 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.522151 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd85777b-rpw8q" event={"ID":"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6","Type":"ContainerDied","Data":"c2a51c895230dde8d9b5e0a162ff9eec9c7d7e7ae4978d8370137369dd0f949b"} Oct 30 13:49:56.595843 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.595820 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:56.713816 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.713778 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") pod \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " Oct 30 13:49:56.714010 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.713881 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") pod \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " Oct 30 13:49:56.714010 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.713987 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") pod \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " Oct 30 13:49:56.714131 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714041 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") pod \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " Oct 30 13:49:56.714131 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714065 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") pod \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " Oct 30 13:49:56.714131 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714094 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp\") pod \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " Oct 30 13:49:56.714131 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714116 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") pod \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\" (UID: \"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6\") " Oct 30 13:49:56.714323 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714228 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config" pod "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:56.714444 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714424 5824 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-config\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.714511 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714436 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp" pod "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:49:56.714565 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714532 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert" pod "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:56.714627 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.714603 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca" pod "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:49:56.716462 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.716433 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert" pod "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:56.716612 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.716465 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg" pod "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6"). InnerVolumeSpecName "kube-api-access-tm9fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:49:56.716612 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.716498 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config" pod "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" (UID: "de19c6ff-20d2-4b1b-a73d-418a7b4fabf6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:49:56.815114 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.815071 5824 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-oauth-serving-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.815114 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.815105 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm9fg\" (UniqueName: \"kubernetes.io/projected/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-kube-api-access-tm9fg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.815114 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.815116 5824 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-tmp\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.815114 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.815125 5824 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.815419 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.815135 5824 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-serving-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.815419 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.815143 5824 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6-console-oauth-config\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:49:56.964534 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:56.964439 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:49:57.527608 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:57.527571 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd85777b-rpw8q" Oct 30 13:49:57.527608 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:57.527592 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd85777b-rpw8q" event={"ID":"de19c6ff-20d2-4b1b-a73d-418a7b4fabf6","Type":"ContainerDied","Data":"3df94a1d75c19881b6612fb5902ea860d3574702a8f4d289f07f476189b2c655"} Oct 30 13:49:57.528087 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:57.527663 5824 scope.go:122] "RemoveContainer" containerID="c2a51c895230dde8d9b5e0a162ff9eec9c7d7e7ae4978d8370137369dd0f949b" Oct 30 13:49:57.585352 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:57.585319 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcd85777b-rpw8q"] Oct 30 13:49:57.598892 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:57.598851 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bcd85777b-rpw8q"] Oct 30 13:49:58.314522 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:58.314488 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" path="/var/lib/kubelet/pods/de19c6ff-20d2-4b1b-a73d-418a7b4fabf6/volumes" Oct 30 13:49:58.718345 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:49:58.718311 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2l4sz" Oct 30 13:50:00.224881 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:50:00.224847 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m5qfs_e9f9b8de-895f-4e27-b089-0f97cfefdff8/serve-healthcheck-canary/0.log" Oct 30 13:50:01.140308 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:50:01.140278 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ldf6b" Oct 30 13:50:03.638815 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:50:03.638773 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:50:33.638220 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:50:33.638167 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:50:43.137379 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:50:43.137318 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:50:43.155000 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:50:43.154964 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:50:43.719959 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:50:43.719926 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 13:51:03.640543 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:51:03.640498 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:51:33.639021 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:51:33.638929 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:52:03.639991 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:52:03.639954 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:52:33.642416 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:52:33.642379 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:53:03.639262 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:53:03.639223 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:53:33.638634 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:53:33.638596 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:54:03.638776 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:54:03.638734 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:54:17.258176 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.258100 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-74mht"] Oct 30 13:54:17.258630 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.258456 5824 cpu_manager.go:414] "containerMap: removing container" podUID="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" containerName="console" Oct 30 13:54:17.258630 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.258469 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" containerName="console" Oct 30 13:54:17.258630 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.258526 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="de19c6ff-20d2-4b1b-a73d-418a7b4fabf6" containerName="console" Oct 30 13:54:17.260619 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.260595 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-74mht" Oct 30 13:54:17.279896 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.279869 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-74mht"] Oct 30 13:54:17.292392 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.292365 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 13:54:17.292523 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.292416 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 13:54:17.292523 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.292416 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dt7vt\"" Oct 30 13:54:17.332860 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.332820 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29100eee-7de7-4441-b5d1-698220226e1b-bound-sa-token\") pod \"cert-manager-597b95cb84-74mht\" (UID: \"29100eee-7de7-4441-b5d1-698220226e1b\") " pod="cert-manager/cert-manager-597b95cb84-74mht" Oct 30 13:54:17.333048 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.332871 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vv4q\" (UniqueName: \"kubernetes.io/projected/29100eee-7de7-4441-b5d1-698220226e1b-kube-api-access-8vv4q\") pod \"cert-manager-597b95cb84-74mht\" (UID: \"29100eee-7de7-4441-b5d1-698220226e1b\") " pod="cert-manager/cert-manager-597b95cb84-74mht" Oct 30 13:54:17.434333 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.434292 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29100eee-7de7-4441-b5d1-698220226e1b-bound-sa-token\") pod \"cert-manager-597b95cb84-74mht\" (UID: \"29100eee-7de7-4441-b5d1-698220226e1b\") " pod="cert-manager/cert-manager-597b95cb84-74mht" Oct 30 13:54:17.434539 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.434344 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vv4q\" (UniqueName: \"kubernetes.io/projected/29100eee-7de7-4441-b5d1-698220226e1b-kube-api-access-8vv4q\") pod \"cert-manager-597b95cb84-74mht\" (UID: \"29100eee-7de7-4441-b5d1-698220226e1b\") " pod="cert-manager/cert-manager-597b95cb84-74mht" Oct 30 13:54:17.460520 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.460484 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29100eee-7de7-4441-b5d1-698220226e1b-bound-sa-token\") pod \"cert-manager-597b95cb84-74mht\" (UID: \"29100eee-7de7-4441-b5d1-698220226e1b\") " pod="cert-manager/cert-manager-597b95cb84-74mht" Oct 30 13:54:17.460781 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.460754 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vv4q\" (UniqueName: \"kubernetes.io/projected/29100eee-7de7-4441-b5d1-698220226e1b-kube-api-access-8vv4q\") pod \"cert-manager-597b95cb84-74mht\" (UID: \"29100eee-7de7-4441-b5d1-698220226e1b\") " pod="cert-manager/cert-manager-597b95cb84-74mht" Oct 30 13:54:17.586789 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.586696 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-74mht" Oct 30 13:54:17.880144 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.880119 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-74mht"] Oct 30 13:54:17.881858 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:54:17.881833 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29100eee_7de7_4441_b5d1_698220226e1b.slice/crio-0f72c4ab6a930ac6bd66c90339603acf1e1897656d68a17aab8f85292eca3779 WatchSource:0}: Error finding container 0f72c4ab6a930ac6bd66c90339603acf1e1897656d68a17aab8f85292eca3779: Status 404 returned error can't find the container with id 0f72c4ab6a930ac6bd66c90339603acf1e1897656d68a17aab8f85292eca3779 Oct 30 13:54:17.883871 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:17.883854 5824 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 13:54:18.484503 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:18.484455 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-74mht" event={"ID":"29100eee-7de7-4441-b5d1-698220226e1b","Type":"ContainerStarted","Data":"0f72c4ab6a930ac6bd66c90339603acf1e1897656d68a17aab8f85292eca3779"} Oct 30 13:54:21.498236 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:21.498201 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-74mht" event={"ID":"29100eee-7de7-4441-b5d1-698220226e1b","Type":"ContainerStarted","Data":"16f9a8f76bcf104f0b9e3ce390ee928046f5374a996262d0b02e243f9c76f17d"} Oct 30 13:54:21.542152 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:21.542100 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-74mht" podStartSLOduration=1.630859546 podStartE2EDuration="4.542083303s" totalImagesPullingTime="2.911223757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:21.541505315 +0000 UTC m=+291.342010770" watchObservedRunningTime="2026-10-30 13:54:21.542083303 +0000 UTC m=+291.342588747" Oct 30 13:54:28.653763 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.653732 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4"] Oct 30 13:54:28.657224 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.657203 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.678135 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.678111 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4"] Oct 30 13:54:28.691138 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.691114 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 13:54:28.691671 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.691657 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 13:54:28.691726 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.691707 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lfdmj\"" Oct 30 13:54:28.691765 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.691736 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 13:54:28.691765 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.691757 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 13:54:28.691948 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.691937 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 13:54:28.743729 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.743703 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/047ccff2-b082-460f-a5c5-33aed3e0886f-cert\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.743846 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.743737 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/047ccff2-b082-460f-a5c5-33aed3e0886f-metrics-cert\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.743894 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.743851 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dcl\" (UniqueName: \"kubernetes.io/projected/047ccff2-b082-460f-a5c5-33aed3e0886f-kube-api-access-k7dcl\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.743894 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.743884 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/047ccff2-b082-460f-a5c5-33aed3e0886f-manager-config\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.844289 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.844260 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dcl\" (UniqueName: \"kubernetes.io/projected/047ccff2-b082-460f-a5c5-33aed3e0886f-kube-api-access-k7dcl\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.844404 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.844305 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/047ccff2-b082-460f-a5c5-33aed3e0886f-manager-config\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.844501 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.844473 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/047ccff2-b082-460f-a5c5-33aed3e0886f-cert\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.844623 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.844529 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/047ccff2-b082-460f-a5c5-33aed3e0886f-metrics-cert\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.845016 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.844994 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/047ccff2-b082-460f-a5c5-33aed3e0886f-manager-config\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.846657 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.846632 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/047ccff2-b082-460f-a5c5-33aed3e0886f-cert\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.846766 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.846723 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/047ccff2-b082-460f-a5c5-33aed3e0886f-metrics-cert\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.870459 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.870440 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dcl\" (UniqueName: \"kubernetes.io/projected/047ccff2-b082-460f-a5c5-33aed3e0886f-kube-api-access-k7dcl\") pod \"lws-controller-manager-778b67f4c5-mdnp4\" (UID: \"047ccff2-b082-460f-a5c5-33aed3e0886f\") " pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:28.967124 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:28.967101 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:29.266234 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:29.266202 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4"] Oct 30 13:54:29.267597 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:54:29.267567 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047ccff2_b082_460f_a5c5_33aed3e0886f.slice/crio-5c58d1cb53f54fcea2b0b7827cbbed623f818540f55b10639d1ec2ea8d20723f WatchSource:0}: Error finding container 5c58d1cb53f54fcea2b0b7827cbbed623f818540f55b10639d1ec2ea8d20723f: Status 404 returned error can't find the container with id 5c58d1cb53f54fcea2b0b7827cbbed623f818540f55b10639d1ec2ea8d20723f Oct 30 13:54:29.526453 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:29.526379 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" event={"ID":"047ccff2-b082-460f-a5c5-33aed3e0886f","Type":"ContainerStarted","Data":"5c58d1cb53f54fcea2b0b7827cbbed623f818540f55b10639d1ec2ea8d20723f"} Oct 30 13:54:32.540781 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:32.540742 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" event={"ID":"047ccff2-b082-460f-a5c5-33aed3e0886f","Type":"ContainerStarted","Data":"2c74024c5820b805a0166a25653e6e9eb9875b6008a349f6ade518aa4286ef7f"} Oct 30 13:54:32.541231 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:32.540923 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:54:32.586787 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:32.586742 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" podStartSLOduration=2.218634263 podStartE2EDuration="4.586729295s" totalImagesPullingTime="2.368095032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:32.586334089 +0000 UTC m=+302.386839538" watchObservedRunningTime="2026-10-30 13:54:32.586729295 +0000 UTC m=+302.387234734" Oct 30 13:54:33.638220 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:54:33.638169 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:54:43.548645 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:54:43.548614 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-778b67f4c5-mdnp4" Oct 30 13:55:03.637397 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:55:03.637359 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:55:30.042831 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.042797 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l"] Oct 30 13:55:30.045356 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.045336 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.061112 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.061089 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l"] Oct 30 13:55:30.104383 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.104354 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b81c6250-d13e-49ae-ab7e-1372263cd4b5-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.104505 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.104398 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.104505 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.104419 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.104589 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.104536 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.104589 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.104554 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81c6250-d13e-49ae-ab7e-1372263cd4b5-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.104589 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.104576 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr46w\" (UniqueName: \"kubernetes.io/projected/b81c6250-d13e-49ae-ab7e-1372263cd4b5-kube-api-access-xr46w\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.149862 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.149836 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 13:55:30.150018 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.149996 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 13:55:30.150086 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.150032 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 13:55:30.150177 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.150160 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 13:55:30.150247 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.150182 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 13:55:30.150335 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.150260 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hnlvr\"" Oct 30 13:55:30.205603 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.205573 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.205717 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.205614 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81c6250-d13e-49ae-ab7e-1372263cd4b5-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.205776 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.205746 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr46w\" (UniqueName: \"kubernetes.io/projected/b81c6250-d13e-49ae-ab7e-1372263cd4b5-kube-api-access-xr46w\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.205832 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.205817 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b81c6250-d13e-49ae-ab7e-1372263cd4b5-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.205903 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.205880 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.205959 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.205907 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.206209 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.206162 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b81c6250-d13e-49ae-ab7e-1372263cd4b5-manifests\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.207960 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.207936 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81c6250-d13e-49ae-ab7e-1372263cd4b5-tmp\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.208245 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.208226 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-metrics-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.208352 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.208330 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.208390 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.208376 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b81c6250-d13e-49ae-ab7e-1372263cd4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.231669 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.231646 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr46w\" (UniqueName: \"kubernetes.io/projected/b81c6250-d13e-49ae-ab7e-1372263cd4b5-kube-api-access-xr46w\") pod \"opendatahub-operator-controller-manager-d55fbd5c9-pnn8l\" (UID: \"b81c6250-d13e-49ae-ab7e-1372263cd4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.332574 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.332519 5824 scope.go:122] "RemoveContainer" containerID="16e70dc35b33f1c481914b0f5597ed058b073e7751d19b541a89db370cacd620" Oct 30 13:55:30.339642 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.339626 5824 scope.go:122] "RemoveContainer" containerID="e2d39a27ba075372648efc479afe802e20ff959aff3bf5479196010a065a141a" Oct 30 13:55:30.346442 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.346425 5824 scope.go:122] "RemoveContainer" containerID="0d90113eda2a6523354a9be7c6a641a0c5cb2942634c809dde2782da5e3c9fca" Oct 30 13:55:30.352653 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.352638 5824 scope.go:122] "RemoveContainer" containerID="9885dc8de5dbb15bdd5d950bec67a2bc7702eb3c391c03c4f8cdeda65a2cc1e0" Oct 30 13:55:30.359834 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.359819 5824 scope.go:122] "RemoveContainer" containerID="de35cd767d5c4c5beee283e34ddc5072a27b59012cca6f940dc013bec5a4d201" Oct 30 13:55:30.366000 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.365959 5824 scope.go:122] "RemoveContainer" containerID="6865aa4d04bdc0cf06400509ee22c9e087684bb2e160896add23c3a1a6255dd3" Oct 30 13:55:30.371994 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.371981 5824 scope.go:122] "RemoveContainer" containerID="3f03b57551d6d4f6f439ad4eac95479e996327e2b3a62befbb59e16ec6aa7551" Oct 30 13:55:30.372675 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.372658 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hnlvr\"" Oct 30 13:55:30.378150 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.378131 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:30.734468 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.734438 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l"] Oct 30 13:55:30.747006 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:55:30.746975 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81c6250_d13e_49ae_ab7e_1372263cd4b5.slice/crio-29ee154c0709eb446f28b653abb0601d3ec4d01ad8b56a83b8315cf8a47e89cf WatchSource:0}: Error finding container 29ee154c0709eb446f28b653abb0601d3ec4d01ad8b56a83b8315cf8a47e89cf: Status 404 returned error can't find the container with id 29ee154c0709eb446f28b653abb0601d3ec4d01ad8b56a83b8315cf8a47e89cf Oct 30 13:55:30.757352 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:30.757320 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" event={"ID":"b81c6250-d13e-49ae-ab7e-1372263cd4b5","Type":"ContainerStarted","Data":"29ee154c0709eb446f28b653abb0601d3ec4d01ad8b56a83b8315cf8a47e89cf"} Oct 30 13:55:33.636452 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:55:33.636420 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:55:33.770949 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:33.770884 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" event={"ID":"b81c6250-d13e-49ae-ab7e-1372263cd4b5","Type":"ContainerStarted","Data":"71862938b9c4e1d50275dbfd7a59df3f7ea2380471c63bb7d8682e880490a3d7"} Oct 30 13:55:34.775382 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:34.775351 5824 generic.go:356] "Generic (PLEG): container finished" podID="b81c6250-d13e-49ae-ab7e-1372263cd4b5" containerID="71862938b9c4e1d50275dbfd7a59df3f7ea2380471c63bb7d8682e880490a3d7" exitCode=0 Oct 30 13:55:34.775797 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:34.775424 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" event={"ID":"b81c6250-d13e-49ae-ab7e-1372263cd4b5","Type":"ContainerDied","Data":"71862938b9c4e1d50275dbfd7a59df3f7ea2380471c63bb7d8682e880490a3d7"} Oct 30 13:55:35.780626 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:35.780591 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" event={"ID":"b81c6250-d13e-49ae-ab7e-1372263cd4b5","Type":"ContainerStarted","Data":"0b82c4c93f8efe9a352da61397d5cb2d6d0f16972b5445f06a77b82ea707a2cb"} Oct 30 13:55:35.781015 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:35.780848 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:35.875510 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:35.875460 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" podStartSLOduration=3.3598729 podStartE2EDuration="6.875448611s" totalImagesPullingTime="3.026012226s" totalInitContainerRuntime="489.563485ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:35.862201227 +0000 UTC m=+365.662706661" watchObservedRunningTime="2026-10-30 13:55:35.875448611 +0000 UTC m=+365.675954054" Oct 30 13:55:46.786661 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:46.786588 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d55fbd5c9-pnn8l" Oct 30 13:55:53.014253 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.014216 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8996685d-dlgtt"] Oct 30 13:55:53.019013 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.018991 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.038764 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.038739 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8996685d-dlgtt"] Oct 30 13:55:53.051939 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.051571 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 13:55:53.052124 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.052092 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 13:55:53.052223 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.052175 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r6bms\"" Oct 30 13:55:53.132284 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.132257 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 13:55:53.132419 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.132308 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 13:55:53.176658 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.176630 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncwt\" (UniqueName: \"kubernetes.io/projected/690e8090-6456-4112-967f-adb1a133841d-kube-api-access-gncwt\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.176782 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.176672 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690e8090-6456-4112-967f-adb1a133841d-tls-certs\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.176782 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.176755 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/690e8090-6456-4112-967f-adb1a133841d-tmp\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.278086 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.278020 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncwt\" (UniqueName: \"kubernetes.io/projected/690e8090-6456-4112-967f-adb1a133841d-kube-api-access-gncwt\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.278086 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.278070 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690e8090-6456-4112-967f-adb1a133841d-tls-certs\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.278275 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.278111 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/690e8090-6456-4112-967f-adb1a133841d-tmp\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.280341 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.280319 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/690e8090-6456-4112-967f-adb1a133841d-tmp\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.280484 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.280467 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690e8090-6456-4112-967f-adb1a133841d-tls-certs\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.303498 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.303473 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncwt\" (UniqueName: \"kubernetes.io/projected/690e8090-6456-4112-967f-adb1a133841d-kube-api-access-gncwt\") pod \"kube-auth-proxy-8996685d-dlgtt\" (UID: \"690e8090-6456-4112-967f-adb1a133841d\") " pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.330881 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.330860 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" Oct 30 13:55:53.623483 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.623461 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8996685d-dlgtt"] Oct 30 13:55:53.625705 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:55:53.625673 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690e8090_6456_4112_967f_adb1a133841d.slice/crio-c60b1d778a582743695196fe727a646a36a8ddcea6b720f73b05bce9ad70f964 WatchSource:0}: Error finding container c60b1d778a582743695196fe727a646a36a8ddcea6b720f73b05bce9ad70f964: Status 404 returned error can't find the container with id c60b1d778a582743695196fe727a646a36a8ddcea6b720f73b05bce9ad70f964 Oct 30 13:55:53.846767 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:53.846731 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" event={"ID":"690e8090-6456-4112-967f-adb1a133841d","Type":"ContainerStarted","Data":"c60b1d778a582743695196fe727a646a36a8ddcea6b720f73b05bce9ad70f964"} Oct 30 13:55:55.855809 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:55.855778 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" event={"ID":"690e8090-6456-4112-967f-adb1a133841d","Type":"ContainerStarted","Data":"e7681e238b468b729f51d0e72bc5a80d0d4f8f8571ac2611fb44df569ed1e2d6"} Oct 30 13:55:55.905083 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:55:55.905044 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8996685d-dlgtt" podStartSLOduration=1.7738503030000001 podStartE2EDuration="3.905031019s" totalImagesPullingTime="2.131180716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:55.904362047 +0000 UTC m=+385.704867489" watchObservedRunningTime="2026-10-30 13:55:55.905031019 +0000 UTC m=+385.705536461" Oct 30 13:56:03.637977 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:56:03.637932 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:56:33.638925 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:56:33.638887 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:56:54.736752 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.736709 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-pvg7l"] Oct 30 13:56:54.739073 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.739049 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.760785 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.760758 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-pvg7l"] Oct 30 13:56:54.784517 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.784481 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 13:56:54.784517 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.784512 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 13:56:54.784713 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.784525 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8jz9d\"" Oct 30 13:56:54.795019 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.794985 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5747106-030e-4cbf-aed2-ff89cf0e457e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.795166 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.795033 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5747106-030e-4cbf-aed2-ff89cf0e457e-cert\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.795166 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.795143 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f528g\" (UniqueName: \"kubernetes.io/projected/c5747106-030e-4cbf-aed2-ff89cf0e457e-kube-api-access-f528g\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.895979 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.895939 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f528g\" (UniqueName: \"kubernetes.io/projected/c5747106-030e-4cbf-aed2-ff89cf0e457e-kube-api-access-f528g\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.896159 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.895996 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5747106-030e-4cbf-aed2-ff89cf0e457e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.896159 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.896045 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5747106-030e-4cbf-aed2-ff89cf0e457e-cert\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.899956 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.899913 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5747106-030e-4cbf-aed2-ff89cf0e457e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.900560 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.900536 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5747106-030e-4cbf-aed2-ff89cf0e457e-cert\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:54.929648 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:54.929610 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f528g\" (UniqueName: \"kubernetes.io/projected/c5747106-030e-4cbf-aed2-ff89cf0e457e-kube-api-access-f528g\") pod \"odh-model-controller-f87fcf56c-pvg7l\" (UID: \"c5747106-030e-4cbf-aed2-ff89cf0e457e\") " pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:55.050351 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:55.050257 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:55.350272 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:55.350243 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-pvg7l"] Oct 30 13:56:55.351444 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:56:55.351419 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5747106_030e_4cbf_aed2_ff89cf0e457e.slice/crio-63c9e68e17d3c30c8fc8aeb494e5c07ba1c216ae6506c74e0450ada82ee718d0 WatchSource:0}: Error finding container 63c9e68e17d3c30c8fc8aeb494e5c07ba1c216ae6506c74e0450ada82ee718d0: Status 404 returned error can't find the container with id 63c9e68e17d3c30c8fc8aeb494e5c07ba1c216ae6506c74e0450ada82ee718d0 Oct 30 13:56:56.084693 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:56.084614 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" event={"ID":"c5747106-030e-4cbf-aed2-ff89cf0e457e","Type":"ContainerStarted","Data":"63c9e68e17d3c30c8fc8aeb494e5c07ba1c216ae6506c74e0450ada82ee718d0"} Oct 30 13:56:58.095508 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:58.095422 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" event={"ID":"c5747106-030e-4cbf-aed2-ff89cf0e457e","Type":"ContainerStarted","Data":"b02bfb36b34cd5717b53bf6698efcfe051d129ad03935d35a141b10a1b3ad364"} Oct 30 13:56:58.095877 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:58.095528 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:56:58.150306 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:56:58.150253 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" podStartSLOduration=1.715355073 podStartE2EDuration="4.150237867s" totalImagesPullingTime="2.434882794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:56:58.149389702 +0000 UTC m=+447.949895146" watchObservedRunningTime="2026-10-30 13:56:58.150237867 +0000 UTC m=+447.950743308" Oct 30 13:57:03.639011 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:57:03.638964 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:57:09.102071 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:57:09.102043 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-pvg7l" Oct 30 13:57:33.641403 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:57:33.641324 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:58:03.637920 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:58:03.637885 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:58:33.640771 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:58:33.640667 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:58:41.206316 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.206236 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs"] Oct 30 13:58:41.209446 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.209425 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" Oct 30 13:58:41.244838 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.244815 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 13:58:41.244838 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.244829 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 13:58:41.245004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.244813 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 13:58:41.245004 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.244899 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-79dfz\"" Oct 30 13:58:41.245756 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.245733 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs"] Oct 30 13:58:41.312401 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.312371 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk68d\" (UniqueName: \"kubernetes.io/projected/d420195b-3cda-4417-88d0-752e1e4018ef-kube-api-access-fk68d\") pod \"dns-operator-controller-manager-65bfd79598-gzmcs\" (UID: \"d420195b-3cda-4417-88d0-752e1e4018ef\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" Oct 30 13:58:41.413342 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.413319 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk68d\" (UniqueName: \"kubernetes.io/projected/d420195b-3cda-4417-88d0-752e1e4018ef-kube-api-access-fk68d\") pod \"dns-operator-controller-manager-65bfd79598-gzmcs\" (UID: \"d420195b-3cda-4417-88d0-752e1e4018ef\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" Oct 30 13:58:41.441008 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.440987 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk68d\" (UniqueName: \"kubernetes.io/projected/d420195b-3cda-4417-88d0-752e1e4018ef-kube-api-access-fk68d\") pod \"dns-operator-controller-manager-65bfd79598-gzmcs\" (UID: \"d420195b-3cda-4417-88d0-752e1e4018ef\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" Oct 30 13:58:41.522332 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.522269 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" Oct 30 13:58:41.822854 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:41.822826 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs"] Oct 30 13:58:41.825039 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:58:41.824998 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd420195b_3cda_4417_88d0_752e1e4018ef.slice/crio-bbf86139a38aed6e540fd5b4e8d90318007b0992538e23d70aa96a22c3ba566f WatchSource:0}: Error finding container bbf86139a38aed6e540fd5b4e8d90318007b0992538e23d70aa96a22c3ba566f: Status 404 returned error can't find the container with id bbf86139a38aed6e540fd5b4e8d90318007b0992538e23d70aa96a22c3ba566f Oct 30 13:58:42.512633 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:42.512551 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" event={"ID":"d420195b-3cda-4417-88d0-752e1e4018ef","Type":"ContainerStarted","Data":"bbf86139a38aed6e540fd5b4e8d90318007b0992538e23d70aa96a22c3ba566f"} Oct 30 13:58:42.922754 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:42.922648 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb"] Oct 30 13:58:42.927407 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:42.927379 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:58:42.961645 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:42.961403 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vsmqt\"" Oct 30 13:58:42.963339 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:42.963251 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb"] Oct 30 13:58:43.028441 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:43.028410 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrwc\" (UniqueName: \"kubernetes.io/projected/8563fad2-d821-45f2-be9b-3e5131b6eb95-kube-api-access-fbrwc\") pod \"limitador-operator-controller-manager-856dd584c5-mk8tb\" (UID: \"8563fad2-d821-45f2-be9b-3e5131b6eb95\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:58:43.129624 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:43.129589 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrwc\" (UniqueName: \"kubernetes.io/projected/8563fad2-d821-45f2-be9b-3e5131b6eb95-kube-api-access-fbrwc\") pod \"limitador-operator-controller-manager-856dd584c5-mk8tb\" (UID: \"8563fad2-d821-45f2-be9b-3e5131b6eb95\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:58:43.156074 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:43.156048 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrwc\" (UniqueName: \"kubernetes.io/projected/8563fad2-d821-45f2-be9b-3e5131b6eb95-kube-api-access-fbrwc\") pod \"limitador-operator-controller-manager-856dd584c5-mk8tb\" (UID: \"8563fad2-d821-45f2-be9b-3e5131b6eb95\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:58:43.246128 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:43.246098 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:58:43.553768 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:43.553736 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb"] Oct 30 13:58:43.555398 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:58:43.555361 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8563fad2_d821_45f2_be9b_3e5131b6eb95.slice/crio-eebd401b27a2b83deed7f2c9aaee3e4dee7368f6d01ad3f7d63685c67d3605b5 WatchSource:0}: Error finding container eebd401b27a2b83deed7f2c9aaee3e4dee7368f6d01ad3f7d63685c67d3605b5: Status 404 returned error can't find the container with id eebd401b27a2b83deed7f2c9aaee3e4dee7368f6d01ad3f7d63685c67d3605b5 Oct 30 13:58:44.523331 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:44.523246 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" event={"ID":"8563fad2-d821-45f2-be9b-3e5131b6eb95","Type":"ContainerStarted","Data":"eebd401b27a2b83deed7f2c9aaee3e4dee7368f6d01ad3f7d63685c67d3605b5"} Oct 30 13:58:44.525289 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:44.525263 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" event={"ID":"d420195b-3cda-4417-88d0-752e1e4018ef","Type":"ContainerStarted","Data":"c4e7a0cb6a7029b7859c569dfe4c569ade6dba4373eeb93ef07a4d8f04e68ab6"} Oct 30 13:58:44.525408 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:44.525368 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" Oct 30 13:58:44.575406 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:44.575358 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" podStartSLOduration=1.218803057 podStartE2EDuration="3.575343708s" totalImagesPullingTime="2.356540651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:44.574111354 +0000 UTC m=+554.374616810" watchObservedRunningTime="2026-10-30 13:58:44.575343708 +0000 UTC m=+554.375849146" Oct 30 13:58:45.531287 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:45.531251 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" event={"ID":"8563fad2-d821-45f2-be9b-3e5131b6eb95","Type":"ContainerStarted","Data":"6c16cec34c4690c35144233aa2ea111e8b498ff1e462077bae8b3a251409be0e"} Oct 30 13:58:45.531472 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:45.531346 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:58:45.576413 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:45.576362 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" podStartSLOduration=2.2541132729999998 podStartE2EDuration="3.576348819s" totalImagesPullingTime="1.322235546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:45.575460388 +0000 UTC m=+555.375965832" watchObservedRunningTime="2026-10-30 13:58:45.576348819 +0000 UTC m=+555.376854266" Oct 30 13:58:55.533601 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:55.533565 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gzmcs" Oct 30 13:58:56.538024 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:58:56.537992 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:59:03.640981 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:59:03.640942 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 13:59:16.475065 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.475030 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb"] Oct 30 13:59:16.475697 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.475299 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" containerName="manager" containerID="cri-o://6c16cec34c4690c35144233aa2ea111e8b498ff1e462077bae8b3a251409be0e" gracePeriod=2 Oct 30 13:59:16.488775 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.488746 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb"] Oct 30 13:59:16.490182 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:59:16.490113 5824 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mk8tb\" is forbidden: User \"system:node:ip-10-0-131-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-200.ec2.internal' and this object" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:59:16.512426 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.512395 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj"] Oct 30 13:59:16.512756 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.512741 5824 cpu_manager.go:414] "containerMap: removing container" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" containerName="manager" Oct 30 13:59:16.512756 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.512755 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" containerName="manager" Oct 30 13:59:16.512892 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.512823 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" containerName="manager" Oct 30 13:59:16.515934 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.515916 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" Oct 30 13:59:16.543944 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.543913 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj"] Oct 30 13:59:16.584863 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:59:16.584831 5824 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mk8tb\" is forbidden: User \"system:node:ip-10-0-131-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-200.ec2.internal' and this object" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:59:16.609984 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.609961 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzqb\" (UniqueName: \"kubernetes.io/projected/49d3309e-57a8-4f1a-92d9-e5521039d3f9-kube-api-access-5gzqb\") pod \"limitador-operator-controller-manager-856dd584c5-chmkj\" (UID: \"49d3309e-57a8-4f1a-92d9-e5521039d3f9\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" Oct 30 13:59:16.661715 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.661689 5824 generic.go:356] "Generic (PLEG): container finished" podID="8563fad2-d821-45f2-be9b-3e5131b6eb95" containerID="6c16cec34c4690c35144233aa2ea111e8b498ff1e462077bae8b3a251409be0e" exitCode=0 Oct 30 13:59:16.710426 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.710405 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzqb\" (UniqueName: \"kubernetes.io/projected/49d3309e-57a8-4f1a-92d9-e5521039d3f9-kube-api-access-5gzqb\") pod \"limitador-operator-controller-manager-856dd584c5-chmkj\" (UID: \"49d3309e-57a8-4f1a-92d9-e5521039d3f9\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" Oct 30 13:59:16.712867 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.712848 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:59:16.728892 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:59:16.728839 5824 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mk8tb\" is forbidden: User \"system:node:ip-10-0-131-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-200.ec2.internal' and this object" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:59:16.740692 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.740670 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzqb\" (UniqueName: \"kubernetes.io/projected/49d3309e-57a8-4f1a-92d9-e5521039d3f9-kube-api-access-5gzqb\") pod \"limitador-operator-controller-manager-856dd584c5-chmkj\" (UID: \"49d3309e-57a8-4f1a-92d9-e5521039d3f9\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" Oct 30 13:59:16.811392 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.811371 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8563fad2-d821-45f2-be9b-3e5131b6eb95-kube-api-access-fbrwc\" (UniqueName: \"kubernetes.io/projected/8563fad2-d821-45f2-be9b-3e5131b6eb95-kube-api-access-fbrwc\") pod \"8563fad2-d821-45f2-be9b-3e5131b6eb95\" (UID: \"8563fad2-d821-45f2-be9b-3e5131b6eb95\") " Oct 30 13:59:16.813250 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.813223 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8563fad2-d821-45f2-be9b-3e5131b6eb95-kube-api-access-fbrwc" pod "8563fad2-d821-45f2-be9b-3e5131b6eb95" (UID: "8563fad2-d821-45f2-be9b-3e5131b6eb95"). InnerVolumeSpecName "kube-api-access-fbrwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:59:16.855072 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.855049 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" Oct 30 13:59:16.912866 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:16.912836 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbrwc\" (UniqueName: \"kubernetes.io/projected/8563fad2-d821-45f2-be9b-3e5131b6eb95-kube-api-access-fbrwc\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 13:59:17.153526 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:17.153503 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj"] Oct 30 13:59:17.155154 ip-10-0-131-200 kubenswrapper[5824]: W1030 13:59:17.155127 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d3309e_57a8_4f1a_92d9_e5521039d3f9.slice/crio-478f1adfbcbaedf9760780c27c86073e8d7cd86b31877f71bed5f24726e4291d WatchSource:0}: Error finding container 478f1adfbcbaedf9760780c27c86073e8d7cd86b31877f71bed5f24726e4291d: Status 404 returned error can't find the container with id 478f1adfbcbaedf9760780c27c86073e8d7cd86b31877f71bed5f24726e4291d Oct 30 13:59:17.536610 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:17.536557 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 13:59:17.667910 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:17.667879 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" event={"ID":"49d3309e-57a8-4f1a-92d9-e5521039d3f9","Type":"ContainerStarted","Data":"ac905eb6da218891c59a7bc2a0ca32790fc03ce996d9a3b6f76b7964d0adbb05"} Oct 30 13:59:17.668100 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:17.667934 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" event={"ID":"49d3309e-57a8-4f1a-92d9-e5521039d3f9","Type":"ContainerStarted","Data":"478f1adfbcbaedf9760780c27c86073e8d7cd86b31877f71bed5f24726e4291d"} Oct 30 13:59:17.668100 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:17.667962 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" Oct 30 13:59:17.669286 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:17.669265 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:59:17.669404 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:17.669271 5824 scope.go:122] "RemoveContainer" containerID="6c16cec34c4690c35144233aa2ea111e8b498ff1e462077bae8b3a251409be0e" Oct 30 13:59:17.715212 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:17.715124 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" podStartSLOduration=1.7151075580000001 podStartE2EDuration="1.715107558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:17.713774709 +0000 UTC m=+587.514280153" watchObservedRunningTime="2026-10-30 13:59:17.715107558 +0000 UTC m=+587.515613003" Oct 30 13:59:17.728389 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:59:17.728347 5824 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mk8tb\" is forbidden: User \"system:node:ip-10-0-131-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-200.ec2.internal' and this object" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:59:17.742993 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:59:17.742953 5824 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mk8tb\" is forbidden: User \"system:node:ip-10-0-131-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-200.ec2.internal' and this object" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mk8tb" Oct 30 13:59:18.314848 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:18.314812 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8563fad2-d821-45f2-be9b-3e5131b6eb95" path="/var/lib/kubelet/pods/8563fad2-d821-45f2-be9b-3e5131b6eb95/volumes" Oct 30 13:59:28.677616 ip-10-0-131-200 kubenswrapper[5824]: I1030 13:59:28.677584 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-chmkj" Oct 30 13:59:33.639562 ip-10-0-131-200 kubenswrapper[5824]: E1030 13:59:33.639522 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:00:02.831061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:02.831020 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfbjz"] Oct 30 14:00:02.834699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:02.834671 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" Oct 30 14:00:02.849445 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:02.849414 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfbjz"] Oct 30 14:00:02.852885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:02.852853 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dj744\"" Oct 30 14:00:02.906654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:02.906625 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22v65\" (UniqueName: \"kubernetes.io/projected/bc47c577-4c3e-4fde-9fd6-513f13d2f444-kube-api-access-22v65\") pod \"authorino-57f7f8659f-wfbjz\" (UID: \"bc47c577-4c3e-4fde-9fd6-513f13d2f444\") " pod="kuadrant-system/authorino-57f7f8659f-wfbjz" Oct 30 14:00:03.007401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.007373 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22v65\" (UniqueName: \"kubernetes.io/projected/bc47c577-4c3e-4fde-9fd6-513f13d2f444-kube-api-access-22v65\") pod \"authorino-57f7f8659f-wfbjz\" (UID: \"bc47c577-4c3e-4fde-9fd6-513f13d2f444\") " pod="kuadrant-system/authorino-57f7f8659f-wfbjz" Oct 30 14:00:03.033912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.033886 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22v65\" (UniqueName: \"kubernetes.io/projected/bc47c577-4c3e-4fde-9fd6-513f13d2f444-kube-api-access-22v65\") pod \"authorino-57f7f8659f-wfbjz\" (UID: \"bc47c577-4c3e-4fde-9fd6-513f13d2f444\") " pod="kuadrant-system/authorino-57f7f8659f-wfbjz" Oct 30 14:00:03.147964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.147904 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" Oct 30 14:00:03.444330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.444294 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfbjz"] Oct 30 14:00:03.445528 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:00:03.445503 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc47c577_4c3e_4fde_9fd6_513f13d2f444.slice/crio-00d9ad44ed7a51736a2a8d422abdc2da6d71f4addc8b4545d7df5df3248dc905 WatchSource:0}: Error finding container 00d9ad44ed7a51736a2a8d422abdc2da6d71f4addc8b4545d7df5df3248dc905: Status 404 returned error can't find the container with id 00d9ad44ed7a51736a2a8d422abdc2da6d71f4addc8b4545d7df5df3248dc905 Oct 30 14:00:03.447110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.447093 5824 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:00:03.469672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.469648 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mplsd"] Oct 30 14:00:03.474211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.474179 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" Oct 30 14:00:03.490910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.490865 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mplsd"] Oct 30 14:00:03.512665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.512642 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2kl\" (UniqueName: \"kubernetes.io/projected/66d5e836-1b87-4a8c-b8ce-8ebc8effa263-kube-api-access-jg2kl\") pod \"authorino-56c6c9ddd5-mplsd\" (UID: \"66d5e836-1b87-4a8c-b8ce-8ebc8effa263\") " pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" Oct 30 14:00:03.613959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.613923 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2kl\" (UniqueName: \"kubernetes.io/projected/66d5e836-1b87-4a8c-b8ce-8ebc8effa263-kube-api-access-jg2kl\") pod \"authorino-56c6c9ddd5-mplsd\" (UID: \"66d5e836-1b87-4a8c-b8ce-8ebc8effa263\") " pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" Oct 30 14:00:03.639613 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:00:03.639573 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:00:03.640474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.640450 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2kl\" (UniqueName: \"kubernetes.io/projected/66d5e836-1b87-4a8c-b8ce-8ebc8effa263-kube-api-access-jg2kl\") pod \"authorino-56c6c9ddd5-mplsd\" (UID: \"66d5e836-1b87-4a8c-b8ce-8ebc8effa263\") " pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" Oct 30 14:00:03.786076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.785989 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" Oct 30 14:00:03.865581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:03.865549 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" event={"ID":"bc47c577-4c3e-4fde-9fd6-513f13d2f444","Type":"ContainerStarted","Data":"00d9ad44ed7a51736a2a8d422abdc2da6d71f4addc8b4545d7df5df3248dc905"} Oct 30 14:00:04.081785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:04.081753 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mplsd"] Oct 30 14:00:04.082639 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:00:04.082607 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d5e836_1b87_4a8c_b8ce_8ebc8effa263.slice/crio-99c783c88456506a35e71a907a195656039c52d8e390e9c47652015600590659 WatchSource:0}: Error finding container 99c783c88456506a35e71a907a195656039c52d8e390e9c47652015600590659: Status 404 returned error can't find the container with id 99c783c88456506a35e71a907a195656039c52d8e390e9c47652015600590659 Oct 30 14:00:04.877532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:04.877496 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" event={"ID":"66d5e836-1b87-4a8c-b8ce-8ebc8effa263","Type":"ContainerStarted","Data":"99c783c88456506a35e71a907a195656039c52d8e390e9c47652015600590659"} Oct 30 14:00:07.891381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:07.891337 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" event={"ID":"bc47c577-4c3e-4fde-9fd6-513f13d2f444","Type":"ContainerStarted","Data":"eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166"} Oct 30 14:00:07.892885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:07.892857 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" event={"ID":"66d5e836-1b87-4a8c-b8ce-8ebc8effa263","Type":"ContainerStarted","Data":"d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c"} Oct 30 14:00:07.939875 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:07.939823 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" podStartSLOduration=2.021754269 podStartE2EDuration="5.939810349s" totalImagesPullingTime="3.91805608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:07.93906648 +0000 UTC m=+637.739571924" watchObservedRunningTime="2026-10-30 14:00:07.939810349 +0000 UTC m=+637.740315792" Oct 30 14:00:07.984763 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:07.984704 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" podStartSLOduration=1.281186061 podStartE2EDuration="4.98468305s" totalImagesPullingTime="3.703496989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:07.983587593 +0000 UTC m=+637.784093036" watchObservedRunningTime="2026-10-30 14:00:07.98468305 +0000 UTC m=+637.785188495" Oct 30 14:00:08.022616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:08.022579 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfbjz"] Oct 30 14:00:09.901573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:09.901528 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" podUID="bc47c577-4c3e-4fde-9fd6-513f13d2f444" containerName="authorino" containerID="cri-o://eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166" gracePeriod=30 Oct 30 14:00:10.139401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.139379 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" Oct 30 14:00:10.169046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.168993 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc47c577-4c3e-4fde-9fd6-513f13d2f444-kube-api-access-22v65\" (UniqueName: \"kubernetes.io/projected/bc47c577-4c3e-4fde-9fd6-513f13d2f444-kube-api-access-22v65\") pod \"bc47c577-4c3e-4fde-9fd6-513f13d2f444\" (UID: \"bc47c577-4c3e-4fde-9fd6-513f13d2f444\") " Oct 30 14:00:10.170942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.170917 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc47c577-4c3e-4fde-9fd6-513f13d2f444-kube-api-access-22v65" pod "bc47c577-4c3e-4fde-9fd6-513f13d2f444" (UID: "bc47c577-4c3e-4fde-9fd6-513f13d2f444"). InnerVolumeSpecName "kube-api-access-22v65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:10.269580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.269544 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22v65\" (UniqueName: \"kubernetes.io/projected/bc47c577-4c3e-4fde-9fd6-513f13d2f444-kube-api-access-22v65\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:00:10.906891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.906850 5824 generic.go:356] "Generic (PLEG): container finished" podID="bc47c577-4c3e-4fde-9fd6-513f13d2f444" containerID="eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166" exitCode=0 Oct 30 14:00:10.907416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.906901 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" Oct 30 14:00:10.907416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.906905 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" event={"ID":"bc47c577-4c3e-4fde-9fd6-513f13d2f444","Type":"ContainerDied","Data":"eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166"} Oct 30 14:00:10.907416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.907019 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wfbjz" event={"ID":"bc47c577-4c3e-4fde-9fd6-513f13d2f444","Type":"ContainerDied","Data":"00d9ad44ed7a51736a2a8d422abdc2da6d71f4addc8b4545d7df5df3248dc905"} Oct 30 14:00:10.907416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.907046 5824 scope.go:122] "RemoveContainer" containerID="eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166" Oct 30 14:00:10.915164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.915132 5824 scope.go:122] "RemoveContainer" containerID="eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166" Oct 30 14:00:10.915457 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:00:10.915435 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166\": container with ID starting with eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166 not found: ID does not exist" containerID="eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166" Oct 30 14:00:10.915521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.915469 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166"} err="failed to get container status \"eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166\": rpc error: code = NotFound desc = could not find container \"eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166\": container with ID starting with eae21a1d28dcb087e8dc273dc038321efa36601ef5add310b7cf2a675fe8b166 not found: ID does not exist" Oct 30 14:00:10.965541 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.965481 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfbjz"] Oct 30 14:00:10.970832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:10.970807 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wfbjz"] Oct 30 14:00:12.314271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:12.314239 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc47c577-4c3e-4fde-9fd6-513f13d2f444" path="/var/lib/kubelet/pods/bc47c577-4c3e-4fde-9fd6-513f13d2f444/volumes" Oct 30 14:00:33.639827 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:00:33.639787 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:00:47.041871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.041843 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vdc6s"] Oct 30 14:00:47.042337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.042173 5824 cpu_manager.go:414] "containerMap: removing container" podUID="bc47c577-4c3e-4fde-9fd6-513f13d2f444" containerName="authorino" Oct 30 14:00:47.042337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.042184 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc47c577-4c3e-4fde-9fd6-513f13d2f444" containerName="authorino" Oct 30 14:00:47.042337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.042273 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc47c577-4c3e-4fde-9fd6-513f13d2f444" containerName="authorino" Oct 30 14:00:47.045113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.045096 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vdc6s" Oct 30 14:00:47.087802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.087774 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vdc6s"] Oct 30 14:00:47.165258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.165225 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849ds\" (UniqueName: \"kubernetes.io/projected/eb91a2d1-a9cd-41de-a3d2-0a70b02d4532-kube-api-access-849ds\") pod \"authorino-6f6944bcdd-vdc6s\" (UID: \"eb91a2d1-a9cd-41de-a3d2-0a70b02d4532\") " pod="kuadrant-system/authorino-6f6944bcdd-vdc6s" Oct 30 14:00:47.252620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.252586 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vdc6s"] Oct 30 14:00:47.252833 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:00:47.252814 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-849ds], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-vdc6s" podUID="eb91a2d1-a9cd-41de-a3d2-0a70b02d4532" Oct 30 14:00:47.265978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.265948 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849ds\" (UniqueName: \"kubernetes.io/projected/eb91a2d1-a9cd-41de-a3d2-0a70b02d4532-kube-api-access-849ds\") pod \"authorino-6f6944bcdd-vdc6s\" (UID: \"eb91a2d1-a9cd-41de-a3d2-0a70b02d4532\") " pod="kuadrant-system/authorino-6f6944bcdd-vdc6s" Oct 30 14:00:47.285258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.285229 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qwtln"] Oct 30 14:00:47.288693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.288673 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:47.294361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.294314 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-849ds\" (UniqueName: \"kubernetes.io/projected/eb91a2d1-a9cd-41de-a3d2-0a70b02d4532-kube-api-access-849ds\") pod \"authorino-6f6944bcdd-vdc6s\" (UID: \"eb91a2d1-a9cd-41de-a3d2-0a70b02d4532\") " pod="kuadrant-system/authorino-6f6944bcdd-vdc6s" Oct 30 14:00:47.307690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.307673 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 14:00:47.318520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.318499 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qwtln"] Oct 30 14:00:47.366989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.366964 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tcq\" (UniqueName: \"kubernetes.io/projected/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-kube-api-access-g2tcq\") pod \"authorino-6889cbd8c4-qwtln\" (UID: \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\") " pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:47.367135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.367023 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-tls-cert\") pod \"authorino-6889cbd8c4-qwtln\" (UID: \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\") " pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:47.399705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.399675 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qwtln"] Oct 30 14:00:47.399914 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:00:47.399891 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g2tcq tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-qwtln" podUID="d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c" Oct 30 14:00:47.442412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.442372 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64bbc5cff7-2xwwl"] Oct 30 14:00:47.445725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.445703 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:00:47.463863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.463836 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64bbc5cff7-2xwwl"] Oct 30 14:00:47.467915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.467891 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-tls-cert\") pod \"authorino-6889cbd8c4-qwtln\" (UID: \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\") " pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:47.468040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.467981 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tcq\" (UniqueName: \"kubernetes.io/projected/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-kube-api-access-g2tcq\") pod \"authorino-6889cbd8c4-qwtln\" (UID: \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\") " pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:47.470730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.470702 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-tls-cert\") pod \"authorino-6889cbd8c4-qwtln\" (UID: \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\") " pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:47.494667 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.494637 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tcq\" (UniqueName: \"kubernetes.io/projected/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-kube-api-access-g2tcq\") pod \"authorino-6889cbd8c4-qwtln\" (UID: \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\") " pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:47.569089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.569016 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c5f5033-acdc-4645-8d09-08da2c00e509-tls-cert\") pod \"authorino-64bbc5cff7-2xwwl\" (UID: \"4c5f5033-acdc-4645-8d09-08da2c00e509\") " pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:00:47.569243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.569126 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frlbg\" (UniqueName: \"kubernetes.io/projected/4c5f5033-acdc-4645-8d09-08da2c00e509-kube-api-access-frlbg\") pod \"authorino-64bbc5cff7-2xwwl\" (UID: \"4c5f5033-acdc-4645-8d09-08da2c00e509\") " pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:00:47.670074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.670046 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c5f5033-acdc-4645-8d09-08da2c00e509-tls-cert\") pod \"authorino-64bbc5cff7-2xwwl\" (UID: \"4c5f5033-acdc-4645-8d09-08da2c00e509\") " pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:00:47.670217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.670117 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frlbg\" (UniqueName: \"kubernetes.io/projected/4c5f5033-acdc-4645-8d09-08da2c00e509-kube-api-access-frlbg\") pod \"authorino-64bbc5cff7-2xwwl\" (UID: \"4c5f5033-acdc-4645-8d09-08da2c00e509\") " pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:00:47.672542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.672517 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c5f5033-acdc-4645-8d09-08da2c00e509-tls-cert\") pod \"authorino-64bbc5cff7-2xwwl\" (UID: \"4c5f5033-acdc-4645-8d09-08da2c00e509\") " pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:00:47.696263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.696174 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frlbg\" (UniqueName: \"kubernetes.io/projected/4c5f5033-acdc-4645-8d09-08da2c00e509-kube-api-access-frlbg\") pod \"authorino-64bbc5cff7-2xwwl\" (UID: \"4c5f5033-acdc-4645-8d09-08da2c00e509\") " pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:00:47.757857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:47.757825 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:00:48.060550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.060522 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64bbc5cff7-2xwwl"] Oct 30 14:00:48.061415 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:00:48.061382 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c5f5033_acdc_4645_8d09_08da2c00e509.slice/crio-2a0c0aa7e6cdd9ced1ac218f28be529bb93796b15111e7edc5161db921c96b90 WatchSource:0}: Error finding container 2a0c0aa7e6cdd9ced1ac218f28be529bb93796b15111e7edc5161db921c96b90: Status 404 returned error can't find the container with id 2a0c0aa7e6cdd9ced1ac218f28be529bb93796b15111e7edc5161db921c96b90 Oct 30 14:00:48.078128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.078108 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vdc6s" Oct 30 14:00:48.078264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.078145 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" event={"ID":"4c5f5033-acdc-4645-8d09-08da2c00e509","Type":"ContainerStarted","Data":"2a0c0aa7e6cdd9ced1ac218f28be529bb93796b15111e7edc5161db921c96b90"} Oct 30 14:00:48.078264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.078213 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:48.086140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.086116 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vdc6s" Oct 30 14:00:48.088158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.088143 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:48.174491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.174468 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb91a2d1-a9cd-41de-a3d2-0a70b02d4532-kube-api-access-849ds\" (UniqueName: \"kubernetes.io/projected/eb91a2d1-a9cd-41de-a3d2-0a70b02d4532-kube-api-access-849ds\") pod \"eb91a2d1-a9cd-41de-a3d2-0a70b02d4532\" (UID: \"eb91a2d1-a9cd-41de-a3d2-0a70b02d4532\") " Oct 30 14:00:48.174602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.174500 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-kube-api-access-g2tcq\" (UniqueName: \"kubernetes.io/projected/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-kube-api-access-g2tcq\") pod \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\" (UID: \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\") " Oct 30 14:00:48.174602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.174555 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-tls-cert\" (UniqueName: \"kubernetes.io/secret/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-tls-cert\") pod \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\" (UID: \"d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c\") " Oct 30 14:00:48.176537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.176511 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-tls-cert" pod "d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c" (UID: "d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:00:48.176666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.176519 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-kube-api-access-g2tcq" pod "d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c" (UID: "d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c"). InnerVolumeSpecName "kube-api-access-g2tcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:48.176666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.176576 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb91a2d1-a9cd-41de-a3d2-0a70b02d4532-kube-api-access-849ds" pod "eb91a2d1-a9cd-41de-a3d2-0a70b02d4532" (UID: "eb91a2d1-a9cd-41de-a3d2-0a70b02d4532"). InnerVolumeSpecName "kube-api-access-849ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:48.276123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.276100 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-849ds\" (UniqueName: \"kubernetes.io/projected/eb91a2d1-a9cd-41de-a3d2-0a70b02d4532-kube-api-access-849ds\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:00:48.276123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.276121 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2tcq\" (UniqueName: \"kubernetes.io/projected/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-kube-api-access-g2tcq\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:00:48.276277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:48.276131 5824 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c-tls-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:00:49.084156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.084114 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qwtln" Oct 30 14:00:49.084156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.084134 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" event={"ID":"4c5f5033-acdc-4645-8d09-08da2c00e509","Type":"ContainerStarted","Data":"2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691"} Oct 30 14:00:49.084707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.084347 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-vdc6s" Oct 30 14:00:49.129629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.129586 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" podStartSLOduration=1.6682345600000001 podStartE2EDuration="2.129575172s" totalImagesPullingTime="461.340612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:49.129395523 +0000 UTC m=+678.929900966" watchObservedRunningTime="2026-10-30 14:00:49.129575172 +0000 UTC m=+678.930080653" Oct 30 14:00:49.158243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.158212 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mplsd"] Oct 30 14:00:49.158427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.158393 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" podUID="66d5e836-1b87-4a8c-b8ce-8ebc8effa263" containerName="authorino" containerID="cri-o://d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c" gracePeriod=30 Oct 30 14:00:49.218070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.218042 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vdc6s"] Oct 30 14:00:49.223481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.223457 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-vdc6s"] Oct 30 14:00:49.305598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.305568 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qwtln"] Oct 30 14:00:49.328147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.328122 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qwtln"] Oct 30 14:00:49.407037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.407014 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" Oct 30 14:00:49.485226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.485177 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d5e836-1b87-4a8c-b8ce-8ebc8effa263-kube-api-access-jg2kl\" (UniqueName: \"kubernetes.io/projected/66d5e836-1b87-4a8c-b8ce-8ebc8effa263-kube-api-access-jg2kl\") pod \"66d5e836-1b87-4a8c-b8ce-8ebc8effa263\" (UID: \"66d5e836-1b87-4a8c-b8ce-8ebc8effa263\") " Oct 30 14:00:49.487285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.487262 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d5e836-1b87-4a8c-b8ce-8ebc8effa263-kube-api-access-jg2kl" pod "66d5e836-1b87-4a8c-b8ce-8ebc8effa263" (UID: "66d5e836-1b87-4a8c-b8ce-8ebc8effa263"). InnerVolumeSpecName "kube-api-access-jg2kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:00:49.585887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:49.585855 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg2kl\" (UniqueName: \"kubernetes.io/projected/66d5e836-1b87-4a8c-b8ce-8ebc8effa263-kube-api-access-jg2kl\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:00:50.089344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.089304 5824 generic.go:356] "Generic (PLEG): container finished" podID="66d5e836-1b87-4a8c-b8ce-8ebc8effa263" containerID="d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c" exitCode=0 Oct 30 14:00:50.089787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.089366 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" Oct 30 14:00:50.089787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.089397 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" event={"ID":"66d5e836-1b87-4a8c-b8ce-8ebc8effa263","Type":"ContainerDied","Data":"d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c"} Oct 30 14:00:50.089787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.089440 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mplsd" event={"ID":"66d5e836-1b87-4a8c-b8ce-8ebc8effa263","Type":"ContainerDied","Data":"99c783c88456506a35e71a907a195656039c52d8e390e9c47652015600590659"} Oct 30 14:00:50.089787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.089463 5824 scope.go:122] "RemoveContainer" containerID="d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c" Oct 30 14:00:50.099501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.099483 5824 scope.go:122] "RemoveContainer" containerID="d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c" Oct 30 14:00:50.099843 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:00:50.099823 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c\": container with ID starting with d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c not found: ID does not exist" containerID="d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c" Oct 30 14:00:50.099931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.099852 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c"} err="failed to get container status \"d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c\": rpc error: code = NotFound desc = could not find container \"d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c\": container with ID starting with d1185bfb896088193c740f77b839d40470947687604efccd4f400c5e65800a5c not found: ID does not exist" Oct 30 14:00:50.151721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.151696 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mplsd"] Oct 30 14:00:50.164790 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.164769 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mplsd"] Oct 30 14:00:50.314491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.314462 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d5e836-1b87-4a8c-b8ce-8ebc8effa263" path="/var/lib/kubelet/pods/66d5e836-1b87-4a8c-b8ce-8ebc8effa263/volumes" Oct 30 14:00:50.314808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.314795 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c" path="/var/lib/kubelet/pods/d9b92a6a-2ebd-4a2e-a1d7-f75136ccbf5c/volumes" Oct 30 14:00:50.315005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:50.314993 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb91a2d1-a9cd-41de-a3d2-0a70b02d4532" path="/var/lib/kubelet/pods/eb91a2d1-a9cd-41de-a3d2-0a70b02d4532/volumes" Oct 30 14:00:56.626930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.626894 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-dwjb5"] Oct 30 14:00:56.627366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.627318 5824 cpu_manager.go:414] "containerMap: removing container" podUID="66d5e836-1b87-4a8c-b8ce-8ebc8effa263" containerName="authorino" Oct 30 14:00:56.627366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.627333 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d5e836-1b87-4a8c-b8ce-8ebc8effa263" containerName="authorino" Oct 30 14:00:56.627437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.627389 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d5e836-1b87-4a8c-b8ce-8ebc8effa263" containerName="authorino" Oct 30 14:00:56.630430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.630412 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.651131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.651108 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-dwjb5"] Oct 30 14:00:56.662905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.662881 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 14:00:56.662905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.662880 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 14:00:56.663069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.662987 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 14:00:56.663069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.663022 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-666kd\"" Oct 30 14:00:56.741305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.741279 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-metrics-certs\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.741404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.741319 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb9cx\" (UniqueName: \"kubernetes.io/projected/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-kube-api-access-vb9cx\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.741448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.741422 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-webhook-cert\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.841984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.841959 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-metrics-certs\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.842081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.841997 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb9cx\" (UniqueName: \"kubernetes.io/projected/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-kube-api-access-vb9cx\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.842081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.842051 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-webhook-cert\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.844564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.844539 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-metrics-certs\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.844660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.844546 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-webhook-cert\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.867915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.867896 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb9cx\" (UniqueName: \"kubernetes.io/projected/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-kube-api-access-vb9cx\") pod \"maas-controller-577f596567-dwjb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.942495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.942476 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:00:56.979038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.979007 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75db85b96b-k2lg7"] Oct 30 14:00:56.982739 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:56.982719 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.010615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.010536 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75db85b96b-k2lg7"] Oct 30 14:00:57.144401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.144371 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvglv\" (UniqueName: \"kubernetes.io/projected/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-kube-api-access-nvglv\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.144401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.144406 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-webhook-cert\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.144599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.144541 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-metrics-certs\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.245956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.245818 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-metrics-certs\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.246311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.245949 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvglv\" (UniqueName: \"kubernetes.io/projected/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-kube-api-access-nvglv\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.246311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.245990 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-webhook-cert\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.248881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.248863 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-webhook-cert\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.248934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.248910 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-metrics-certs\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.249218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.249185 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-dwjb5"] Oct 30 14:00:57.250809 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:00:57.250787 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbd0c5f_b4f9_43d8_adda_3c0ca96f9cb5.slice/crio-5210a357f44053dfe8bc69ad4039623ebd40df0f6b799b51107e8c34304dd5dc WatchSource:0}: Error finding container 5210a357f44053dfe8bc69ad4039623ebd40df0f6b799b51107e8c34304dd5dc: Status 404 returned error can't find the container with id 5210a357f44053dfe8bc69ad4039623ebd40df0f6b799b51107e8c34304dd5dc Oct 30 14:00:57.272277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.272256 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvglv\" (UniqueName: \"kubernetes.io/projected/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-kube-api-access-nvglv\") pod \"maas-controller-75db85b96b-k2lg7\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.300105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.300082 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:00:57.599256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:57.599229 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75db85b96b-k2lg7"] Oct 30 14:00:57.600388 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:00:57.600362 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0be2a8_6c76_45c9_b301_4e40d5aad04d.slice/crio-f9109701e3b983d7d2108818077eb0dc03b1fff68c93f3d83624731b4e36029f WatchSource:0}: Error finding container f9109701e3b983d7d2108818077eb0dc03b1fff68c93f3d83624731b4e36029f: Status 404 returned error can't find the container with id f9109701e3b983d7d2108818077eb0dc03b1fff68c93f3d83624731b4e36029f Oct 30 14:00:58.129385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:58.129299 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-dwjb5" event={"ID":"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5","Type":"ContainerStarted","Data":"5210a357f44053dfe8bc69ad4039623ebd40df0f6b799b51107e8c34304dd5dc"} Oct 30 14:00:58.139219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:00:58.131958 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-k2lg7" event={"ID":"1c0be2a8-6c76-45c9-b301-4e40d5aad04d","Type":"ContainerStarted","Data":"f9109701e3b983d7d2108818077eb0dc03b1fff68c93f3d83624731b4e36029f"} Oct 30 14:01:01.148273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:01.148237 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-k2lg7" event={"ID":"1c0be2a8-6c76-45c9-b301-4e40d5aad04d","Type":"ContainerStarted","Data":"5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1"} Oct 30 14:01:01.148786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:01.148335 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:01:01.149823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:01.149793 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-dwjb5" event={"ID":"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5","Type":"ContainerStarted","Data":"99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e"} Oct 30 14:01:01.149949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:01.149852 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:01:01.198304 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:01.198251 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75db85b96b-k2lg7" podStartSLOduration=2.478037944 podStartE2EDuration="5.198236255s" totalImagesPullingTime="2.720198311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:01:01.196418519 +0000 UTC m=+690.996923963" watchObservedRunningTime="2026-10-30 14:01:01.198236255 +0000 UTC m=+690.998741699" Oct 30 14:01:01.245581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:01.245525 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-dwjb5" podStartSLOduration=2.359300765 podStartE2EDuration="5.245508324s" totalImagesPullingTime="2.886207559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:01:01.245035794 +0000 UTC m=+691.045541238" watchObservedRunningTime="2026-10-30 14:01:01.245508324 +0000 UTC m=+691.046013770" Oct 30 14:01:03.639681 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:01:03.639635 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:01:04.006756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.006722 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw"] Oct 30 14:01:04.010413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.010392 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.042091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.042058 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 14:01:04.042338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.042127 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 14:01:04.042338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.042135 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 14:01:04.042338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.042152 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-nntkz\"" Oct 30 14:01:04.043176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.043156 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw"] Oct 30 14:01:04.113254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.113216 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c048d8ea-ea88-4cbd-ac80-892f46baa62d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.113411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.113343 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c048d8ea-ea88-4cbd-ac80-892f46baa62d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.113457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.113440 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c048d8ea-ea88-4cbd-ac80-892f46baa62d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.113492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.113469 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zlt\" (UniqueName: \"kubernetes.io/projected/c048d8ea-ea88-4cbd-ac80-892f46baa62d-kube-api-access-z7zlt\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.214393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.214357 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c048d8ea-ea88-4cbd-ac80-892f46baa62d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.214393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.214394 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zlt\" (UniqueName: \"kubernetes.io/projected/c048d8ea-ea88-4cbd-ac80-892f46baa62d-kube-api-access-z7zlt\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.214603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.214435 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c048d8ea-ea88-4cbd-ac80-892f46baa62d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.214603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.214481 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c048d8ea-ea88-4cbd-ac80-892f46baa62d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.215134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.215107 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c048d8ea-ea88-4cbd-ac80-892f46baa62d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.215333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.215311 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c048d8ea-ea88-4cbd-ac80-892f46baa62d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.217021 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.216998 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c048d8ea-ea88-4cbd-ac80-892f46baa62d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.241272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.241251 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zlt\" (UniqueName: \"kubernetes.io/projected/c048d8ea-ea88-4cbd-ac80-892f46baa62d-kube-api-access-z7zlt\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-q2dfw\" (UID: \"c048d8ea-ea88-4cbd-ac80-892f46baa62d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.323001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.322929 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:04.622126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:04.622098 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw"] Oct 30 14:01:04.623430 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:01:04.623405 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc048d8ea_ea88_4cbd_ac80_892f46baa62d.slice/crio-8cc7b193c038e104e7f01309604f83847a3c2144fe8350f295323070d341b9b9 WatchSource:0}: Error finding container 8cc7b193c038e104e7f01309604f83847a3c2144fe8350f295323070d341b9b9: Status 404 returned error can't find the container with id 8cc7b193c038e104e7f01309604f83847a3c2144fe8350f295323070d341b9b9 Oct 30 14:01:05.169499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:05.169464 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" event={"ID":"c048d8ea-ea88-4cbd-ac80-892f46baa62d","Type":"ContainerStarted","Data":"8cc7b193c038e104e7f01309604f83847a3c2144fe8350f295323070d341b9b9"} Oct 30 14:01:12.158491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.158462 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:01:12.159128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.159109 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:01:12.261816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.261782 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-dwjb5"] Oct 30 14:01:12.262037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.261995 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-dwjb5" podUID="fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" containerName="manager" containerID="cri-o://99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e" gracePeriod=30 Oct 30 14:01:12.512287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.512262 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:01:12.596021 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.595985 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-webhook-cert\") pod \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " Oct 30 14:01:12.596247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.596066 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-kube-api-access-vb9cx\" (UniqueName: \"kubernetes.io/projected/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-kube-api-access-vb9cx\") pod \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " Oct 30 14:01:12.596247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.596135 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-metrics-certs\") pod \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\" (UID: \"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5\") " Oct 30 14:01:12.598731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.598702 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-webhook-cert" pod "fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" (UID: "fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:01:12.598861 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.598748 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-kube-api-access-vb9cx" pod "fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" (UID: "fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5"). InnerVolumeSpecName "kube-api-access-vb9cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:01:12.601614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.601592 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-metrics-certs" pod "fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" (UID: "fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:01:12.697535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.697467 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb9cx\" (UniqueName: \"kubernetes.io/projected/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-kube-api-access-vb9cx\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:01:12.697535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.697497 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:01:12.697535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:12.697507 5824 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5-webhook-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:01:13.213506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.213467 5824 generic.go:356] "Generic (PLEG): container finished" podID="fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" containerID="99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e" exitCode=0 Oct 30 14:01:13.213966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.213528 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-dwjb5" event={"ID":"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5","Type":"ContainerDied","Data":"99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e"} Oct 30 14:01:13.213966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.213586 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-dwjb5" event={"ID":"fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5","Type":"ContainerDied","Data":"5210a357f44053dfe8bc69ad4039623ebd40df0f6b799b51107e8c34304dd5dc"} Oct 30 14:01:13.213966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.213616 5824 scope.go:122] "RemoveContainer" containerID="99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e" Oct 30 14:01:13.213966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.213530 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-dwjb5" Oct 30 14:01:13.224107 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.224088 5824 scope.go:122] "RemoveContainer" containerID="99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e" Oct 30 14:01:13.224417 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:01:13.224393 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e\": container with ID starting with 99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e not found: ID does not exist" containerID="99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e" Oct 30 14:01:13.224517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.224426 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e"} err="failed to get container status \"99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e\": rpc error: code = NotFound desc = could not find container \"99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e\": container with ID starting with 99e0228d9148c503b3fa76ec9579f16d98a16556b057d3aa0826d1c42617746e not found: ID does not exist" Oct 30 14:01:13.272059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.272032 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-dwjb5"] Oct 30 14:01:13.289995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:13.289957 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-dwjb5"] Oct 30 14:01:14.315339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:14.315304 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" path="/var/lib/kubelet/pods/fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5/volumes" Oct 30 14:01:16.229375 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:16.229341 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" event={"ID":"c048d8ea-ea88-4cbd-ac80-892f46baa62d","Type":"ContainerStarted","Data":"c56f838594376e0c66f34c845e214cb3a91aef4bdb94dfe80c7bec585ff8d5e7"} Oct 30 14:01:16.229730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:16.229467 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:27.238781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:27.238746 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" Oct 30 14:01:27.286860 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:01:27.286814 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-q2dfw" podStartSLOduration=13.660442858 podStartE2EDuration="24.286798032s" totalImagesPullingTime="10.626355174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:01:16.289997636 +0000 UTC m=+706.090503080" watchObservedRunningTime="2026-10-30 14:01:27.286798032 +0000 UTC m=+717.087303477" Oct 30 14:01:33.640380 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:01:33.640342 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:02:03.637907 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:02:03.637871 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:02:33.638447 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:02:33.638414 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:02:40.138157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.138125 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9"] Oct 30 14:02:40.138611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.138520 5824 cpu_manager.go:414] "containerMap: removing container" podUID="fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" containerName="manager" Oct 30 14:02:40.138611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.138533 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" containerName="manager" Oct 30 14:02:40.138611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.138589 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbd0c5f-b4f9-43d8-adda-3c0ca96f9cb5" containerName="manager" Oct 30 14:02:40.141384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.141362 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.162630 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.162596 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 14:02:40.163576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.163557 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9"] Oct 30 14:02:40.175996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.175974 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lmhz8\"" Oct 30 14:02:40.176233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.176199 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 14:02:40.262477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.262454 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 14:02:40.320287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.320265 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.320393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.320306 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.320393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.320328 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.320514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.320393 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.320514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.320451 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c93f2a70-0763-42c9-b922-b1bec73d6a05-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.320617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.320566 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfglg\" (UniqueName: \"kubernetes.io/projected/c93f2a70-0763-42c9-b922-b1bec73d6a05-kube-api-access-jfglg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421071 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfglg\" (UniqueName: \"kubernetes.io/projected/c93f2a70-0763-42c9-b922-b1bec73d6a05-kube-api-access-jfglg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421139 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421230 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421265 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421290 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421327 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c93f2a70-0763-42c9-b922-b1bec73d6a05-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421662 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421743 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.421795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.421760 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.423400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.423376 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c93f2a70-0763-42c9-b922-b1bec73d6a05-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.423642 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.423624 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c93f2a70-0763-42c9-b922-b1bec73d6a05-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.448764 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.448741 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfglg\" (UniqueName: \"kubernetes.io/projected/c93f2a70-0763-42c9-b922-b1bec73d6a05-kube-api-access-jfglg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-84kw9\" (UID: \"c93f2a70-0763-42c9-b922-b1bec73d6a05\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.456648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.456626 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:40.772010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:40.771979 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9"] Oct 30 14:02:40.773117 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:02:40.773093 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93f2a70_0763_42c9_b922_b1bec73d6a05.slice/crio-e38d08dadde0d541a9ec96dc8a190513f9d15c63c708ea21c6b91b1a31b0ddc4 WatchSource:0}: Error finding container e38d08dadde0d541a9ec96dc8a190513f9d15c63c708ea21c6b91b1a31b0ddc4: Status 404 returned error can't find the container with id e38d08dadde0d541a9ec96dc8a190513f9d15c63c708ea21c6b91b1a31b0ddc4 Oct 30 14:02:41.580309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:41.580261 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" event={"ID":"c93f2a70-0763-42c9-b922-b1bec73d6a05","Type":"ContainerStarted","Data":"e38d08dadde0d541a9ec96dc8a190513f9d15c63c708ea21c6b91b1a31b0ddc4"} Oct 30 14:02:42.586028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:42.585995 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" event={"ID":"c93f2a70-0763-42c9-b922-b1bec73d6a05","Type":"ContainerStarted","Data":"3b9cea33758ce6b125b76afa199466c544f231c43c713011cc4b3e0ccaef2f98"} Oct 30 14:02:42.632753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:42.632710 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" podStartSLOduration=1.449413123 podStartE2EDuration="2.632696964s" totalImagesPullingTime="1.183283841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:42.631160966 +0000 UTC m=+792.431666393" watchObservedRunningTime="2026-10-30 14:02:42.632696964 +0000 UTC m=+792.433202407" Oct 30 14:02:43.452837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.452802 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv"] Oct 30 14:02:43.455396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.455372 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.457789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.457766 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:43.478244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.478220 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv"] Oct 30 14:02:43.486702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.486671 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 14:02:43.556938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.556901 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6xb\" (UniqueName: \"kubernetes.io/projected/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-kube-api-access-bx6xb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.557123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.556943 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.557123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.556973 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.557288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.557163 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.557288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.557243 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.557396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.557315 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.659211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.658636 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.659211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.658716 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6xb\" (UniqueName: \"kubernetes.io/projected/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-kube-api-access-bx6xb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.659211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.658747 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.659211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.658781 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.659211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.658839 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.659211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.658897 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.659936 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.659359 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.665222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.662230 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.665222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.662790 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.671504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.671471 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.671703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.671681 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.689013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.688972 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6xb\" (UniqueName: \"kubernetes.io/projected/ad920ab1-dd5c-4167-b255-c68d6b9b8bc4-kube-api-access-bx6xb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-c5gqv\" (UID: \"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:43.770450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:43.770362 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:44.076471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:44.076444 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv"] Oct 30 14:02:44.077476 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:02:44.077449 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad920ab1_dd5c_4167_b255_c68d6b9b8bc4.slice/crio-870d5777ace51785025c3c20da283301dee308be037daac9d467c5463d1fb6bc WatchSource:0}: Error finding container 870d5777ace51785025c3c20da283301dee308be037daac9d467c5463d1fb6bc: Status 404 returned error can't find the container with id 870d5777ace51785025c3c20da283301dee308be037daac9d467c5463d1fb6bc Oct 30 14:02:44.596721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:44.596651 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" event={"ID":"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4","Type":"ContainerStarted","Data":"e9afda116aa14844a5ec8a9bdff224455dc107a1a7719f0bd63036ae590fd8da"} Oct 30 14:02:44.596721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:44.596700 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" event={"ID":"ad920ab1-dd5c-4167-b255-c68d6b9b8bc4","Type":"ContainerStarted","Data":"870d5777ace51785025c3c20da283301dee308be037daac9d467c5463d1fb6bc"} Oct 30 14:02:44.643544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:44.643497 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" podStartSLOduration=1.450325214 podStartE2EDuration="1.643485009s" totalImagesPullingTime="193.159795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:44.642593407 +0000 UTC m=+794.443098862" watchObservedRunningTime="2026-10-30 14:02:44.643485009 +0000 UTC m=+794.443990452" Oct 30 14:02:44.771204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:44.771163 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:46.470126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.470088 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2"] Oct 30 14:02:46.472952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.472928 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.482478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.482451 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.482628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.482485 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkc6\" (UniqueName: \"kubernetes.io/projected/d6a5964c-caae-4580-9c21-03a9c53f86b9-kube-api-access-7zkc6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.482628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.482600 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a5964c-caae-4580-9c21-03a9c53f86b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.483013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.482671 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.483013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.482697 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.483013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.482761 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.495632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.495602 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2"] Oct 30 14:02:46.504824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.504802 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 14:02:46.583652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.583596 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a5964c-caae-4580-9c21-03a9c53f86b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.583802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.583661 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.583802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.583692 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.583802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.583743 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.583971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.583798 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.583971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.583830 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkc6\" (UniqueName: \"kubernetes.io/projected/d6a5964c-caae-4580-9c21-03a9c53f86b9-kube-api-access-7zkc6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.584171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.584145 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.584278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.584157 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.584513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.584488 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.586098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.586078 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a5964c-caae-4580-9c21-03a9c53f86b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.586177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.586160 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a5964c-caae-4580-9c21-03a9c53f86b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.611896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.611866 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkc6\" (UniqueName: \"kubernetes.io/projected/d6a5964c-caae-4580-9c21-03a9c53f86b9-kube-api-access-7zkc6\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2\" (UID: \"d6a5964c-caae-4580-9c21-03a9c53f86b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:46.787656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:46.787591 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:47.094106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:47.094076 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2"] Oct 30 14:02:47.095049 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:02:47.095013 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a5964c_caae_4580_9c21_03a9c53f86b9.slice/crio-07288a69cf10c6378188b2904404e8470cc14ba3c3a2b2d695ae6f39bfb39eb5 WatchSource:0}: Error finding container 07288a69cf10c6378188b2904404e8470cc14ba3c3a2b2d695ae6f39bfb39eb5: Status 404 returned error can't find the container with id 07288a69cf10c6378188b2904404e8470cc14ba3c3a2b2d695ae6f39bfb39eb5 Oct 30 14:02:47.613216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:47.613124 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" event={"ID":"d6a5964c-caae-4580-9c21-03a9c53f86b9","Type":"ContainerStarted","Data":"59cfbc64301143185eb8e2d28efe07c7acda20bbca64317dd05fea33a3d0376b"} Oct 30 14:02:47.613216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:47.613212 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" event={"ID":"d6a5964c-caae-4580-9c21-03a9c53f86b9","Type":"ContainerStarted","Data":"07288a69cf10c6378188b2904404e8470cc14ba3c3a2b2d695ae6f39bfb39eb5"} Oct 30 14:02:47.788505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:47.788472 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:48.349500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.349430 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" podStartSLOduration=2.149253705 podStartE2EDuration="2.349414493s" totalImagesPullingTime="200.160788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:47.667583688 +0000 UTC m=+797.468089132" watchObservedRunningTime="2026-10-30 14:02:48.349414493 +0000 UTC m=+798.149919915" Oct 30 14:02:48.350499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.350473 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz"] Oct 30 14:02:48.353063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.353045 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.386672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.386620 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 14:02:48.387917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.387891 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz"] Oct 30 14:02:48.397283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.397256 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.397402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.397307 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.397533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.397514 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wc2\" (UniqueName: \"kubernetes.io/projected/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-kube-api-access-x5wc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.397603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.397580 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.397687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.397673 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.397743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.397723 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498165 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498233 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498268 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wc2\" (UniqueName: \"kubernetes.io/projected/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-kube-api-access-x5wc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498308 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498338 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498383 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498755 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498799 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.498880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.498838 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.500444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.500427 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.500781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.500764 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.524358 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.524329 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wc2\" (UniqueName: \"kubernetes.io/projected/7c410a7f-e3b9-485d-bbd4-2b3d83c2d297-kube-api-access-x5wc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz\" (UID: \"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.668717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.668621 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:48.972927 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:48.972894 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz"] Oct 30 14:02:48.973380 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:02:48.973351 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c410a7f_e3b9_485d_bbd4_2b3d83c2d297.slice/crio-85b884914b1efe5a8faea13283106560d52c4af741a260da551b6dd0f05918b8 WatchSource:0}: Error finding container 85b884914b1efe5a8faea13283106560d52c4af741a260da551b6dd0f05918b8: Status 404 returned error can't find the container with id 85b884914b1efe5a8faea13283106560d52c4af741a260da551b6dd0f05918b8 Oct 30 14:02:49.623836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:49.623743 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" event={"ID":"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297","Type":"ContainerStarted","Data":"b12beef2cb88060e83c7b34e1383e87f0e134ef1f63294ea9a47df38562dfa59"} Oct 30 14:02:49.623836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:49.623789 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" event={"ID":"7c410a7f-e3b9-485d-bbd4-2b3d83c2d297","Type":"ContainerStarted","Data":"85b884914b1efe5a8faea13283106560d52c4af741a260da551b6dd0f05918b8"} Oct 30 14:02:49.669409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:49.669371 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:49.672066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:49.672017 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podStartSLOduration=1.4532150289999999 podStartE2EDuration="1.672001438s" totalImagesPullingTime="218.786409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:49.671478955 +0000 UTC m=+799.471984396" watchObservedRunningTime="2026-10-30 14:02:49.672001438 +0000 UTC m=+799.472506895" Oct 30 14:02:50.457702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:50.457666 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:50.470033 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:50.470010 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:50.638427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:50.638405 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-84kw9" Oct 30 14:02:53.771413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:53.771381 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:53.783436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:53.783414 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:54.657519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:54.657492 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-c5gqv" Oct 30 14:02:56.787757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:56.787720 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:56.799732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:56.799711 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:57.671472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:57.671442 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2" Oct 30 14:02:58.669536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:58.669505 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:58.682034 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:58.682008 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:02:59.679351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:02:59.679325 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:03:02.265687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.265654 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72"] Oct 30 14:03:02.268415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.268397 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.287957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.287938 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 14:03:02.305856 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.305539 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72"] Oct 30 14:03:02.318411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318389 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.318503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318418 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.318503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318440 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhqs\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-kube-api-access-9hhqs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.318622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318601 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.318677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318660 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.318717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318689 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.318809 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318790 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.318884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318870 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.318937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.318902 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.373750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.373726 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5drtm\"" Oct 30 14:03:02.419815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419794 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.419899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419821 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.419899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419841 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.419899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419857 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.419899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419880 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhqs\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-kube-api-access-9hhqs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419917 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419936 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419955 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.419999 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.420333 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.420376 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.420453 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.420578 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.420653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.420605 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.422386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.422366 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.422491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.422474 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.446396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.446369 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.447585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.447563 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhqs\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-kube-api-access-9hhqs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6cf72\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.579995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.579947 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:02.904460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:02.904433 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72"] Oct 30 14:03:02.904879 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:03:02.904854 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aef07d_fc7a_4a47_8a50_4a35d9f19a78.slice/crio-50b1976367b045aae7a1e17d9e6c12445a6b670dcba047dfb034e2f1eccafe47 WatchSource:0}: Error finding container 50b1976367b045aae7a1e17d9e6c12445a6b670dcba047dfb034e2f1eccafe47: Status 404 returned error can't find the container with id 50b1976367b045aae7a1e17d9e6c12445a6b670dcba047dfb034e2f1eccafe47 Oct 30 14:03:03.341513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.341481 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cd6b76db-pbblr"] Oct 30 14:03:03.344634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.344612 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cd6b76db-pbblr" Oct 30 14:03:03.367879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.367857 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cd6b76db-pbblr"] Oct 30 14:03:03.428154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.428119 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhpkf\" (UniqueName: \"kubernetes.io/projected/3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e-kube-api-access-rhpkf\") pod \"authorino-5cd6b76db-pbblr\" (UID: \"3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e\") " pod="kuadrant-system/authorino-5cd6b76db-pbblr" Oct 30 14:03:03.428335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.428319 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e-tls-cert\") pod \"authorino-5cd6b76db-pbblr\" (UID: \"3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e\") " pod="kuadrant-system/authorino-5cd6b76db-pbblr" Oct 30 14:03:03.529399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.529375 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhpkf\" (UniqueName: \"kubernetes.io/projected/3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e-kube-api-access-rhpkf\") pod \"authorino-5cd6b76db-pbblr\" (UID: \"3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e\") " pod="kuadrant-system/authorino-5cd6b76db-pbblr" Oct 30 14:03:03.529484 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.529447 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e-tls-cert\") pod \"authorino-5cd6b76db-pbblr\" (UID: \"3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e\") " pod="kuadrant-system/authorino-5cd6b76db-pbblr" Oct 30 14:03:03.531772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.531747 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e-tls-cert\") pod \"authorino-5cd6b76db-pbblr\" (UID: \"3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e\") " pod="kuadrant-system/authorino-5cd6b76db-pbblr" Oct 30 14:03:03.555705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.555684 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhpkf\" (UniqueName: \"kubernetes.io/projected/3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e-kube-api-access-rhpkf\") pod \"authorino-5cd6b76db-pbblr\" (UID: \"3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e\") " pod="kuadrant-system/authorino-5cd6b76db-pbblr" Oct 30 14:03:03.642763 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:03:03.642689 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:03:03.655262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.655238 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cd6b76db-pbblr" Oct 30 14:03:03.687690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.687658 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" event={"ID":"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78","Type":"ContainerStarted","Data":"50b1976367b045aae7a1e17d9e6c12445a6b670dcba047dfb034e2f1eccafe47"} Oct 30 14:03:03.955249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:03.955222 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cd6b76db-pbblr"] Oct 30 14:03:04.696085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:04.696054 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cd6b76db-pbblr" event={"ID":"3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e","Type":"ContainerStarted","Data":"6a13cf75f187913ab767730007624ad69d643405769d6e615596213154c31f8a"} Oct 30 14:03:04.696515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:04.696099 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cd6b76db-pbblr" event={"ID":"3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e","Type":"ContainerStarted","Data":"857cb124046c9dea6802bc4466b6dad8f690db12beb58e525455a2cabb3573c2"} Oct 30 14:03:04.741789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:04.741695 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5cd6b76db-pbblr" podStartSLOduration=1.253732798 podStartE2EDuration="1.741678149s" totalImagesPullingTime="487.945351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:04.740464266 +0000 UTC m=+814.540969710" watchObservedRunningTime="2026-10-30 14:03:04.741678149 +0000 UTC m=+814.542183593" Oct 30 14:03:04.769505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:04.769185 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64bbc5cff7-2xwwl"] Oct 30 14:03:04.769505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:04.769481 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" podUID="4c5f5033-acdc-4645-8d09-08da2c00e509" containerName="authorino" containerID="cri-o://2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691" gracePeriod=30 Oct 30 14:03:05.032024 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.031998 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:03:05.145795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.145757 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5f5033-acdc-4645-8d09-08da2c00e509-kube-api-access-frlbg\" (UniqueName: \"kubernetes.io/projected/4c5f5033-acdc-4645-8d09-08da2c00e509-kube-api-access-frlbg\") pod \"4c5f5033-acdc-4645-8d09-08da2c00e509\" (UID: \"4c5f5033-acdc-4645-8d09-08da2c00e509\") " Oct 30 14:03:05.145969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.145838 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5f5033-acdc-4645-8d09-08da2c00e509-tls-cert\" (UniqueName: \"kubernetes.io/secret/4c5f5033-acdc-4645-8d09-08da2c00e509-tls-cert\") pod \"4c5f5033-acdc-4645-8d09-08da2c00e509\" (UID: \"4c5f5033-acdc-4645-8d09-08da2c00e509\") " Oct 30 14:03:05.148404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.148369 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5f5033-acdc-4645-8d09-08da2c00e509-kube-api-access-frlbg" pod "4c5f5033-acdc-4645-8d09-08da2c00e509" (UID: "4c5f5033-acdc-4645-8d09-08da2c00e509"). InnerVolumeSpecName "kube-api-access-frlbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:03:05.157525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.157498 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5f5033-acdc-4645-8d09-08da2c00e509-tls-cert" pod "4c5f5033-acdc-4645-8d09-08da2c00e509" (UID: "4c5f5033-acdc-4645-8d09-08da2c00e509"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:03:05.246934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.246896 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frlbg\" (UniqueName: \"kubernetes.io/projected/4c5f5033-acdc-4645-8d09-08da2c00e509-kube-api-access-frlbg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:03:05.246934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.246932 5824 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c5f5033-acdc-4645-8d09-08da2c00e509-tls-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:03:05.712236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.712108 5824 generic.go:356] "Generic (PLEG): container finished" podID="4c5f5033-acdc-4645-8d09-08da2c00e509" containerID="2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691" exitCode=0 Oct 30 14:03:05.712727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.712233 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" event={"ID":"4c5f5033-acdc-4645-8d09-08da2c00e509","Type":"ContainerDied","Data":"2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691"} Oct 30 14:03:05.712727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.712271 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" Oct 30 14:03:05.712727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.712285 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64bbc5cff7-2xwwl" event={"ID":"4c5f5033-acdc-4645-8d09-08da2c00e509","Type":"ContainerDied","Data":"2a0c0aa7e6cdd9ced1ac218f28be529bb93796b15111e7edc5161db921c96b90"} Oct 30 14:03:05.712727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.712311 5824 scope.go:122] "RemoveContainer" containerID="2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691" Oct 30 14:03:05.731290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.731265 5824 scope.go:122] "RemoveContainer" containerID="2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691" Oct 30 14:03:05.731679 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:03:05.731650 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691\": container with ID starting with 2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691 not found: ID does not exist" containerID="2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691" Oct 30 14:03:05.731781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.731692 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691"} err="failed to get container status \"2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691\": rpc error: code = NotFound desc = could not find container \"2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691\": container with ID starting with 2a549db41199ee566e2e8c9c3392cfaa5e9669dac9fab0764cfcc7d532a9f691 not found: ID does not exist" Oct 30 14:03:05.770811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.770783 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64bbc5cff7-2xwwl"] Oct 30 14:03:05.784434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.784407 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64bbc5cff7-2xwwl"] Oct 30 14:03:05.911864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.911825 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:05.911978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.911894 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:05.911978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:05.911926 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:06.316382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:06.316287 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5f5033-acdc-4645-8d09-08da2c00e509" path="/var/lib/kubelet/pods/4c5f5033-acdc-4645-8d09-08da2c00e509/volumes" Oct 30 14:03:06.721230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:06.721180 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" event={"ID":"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78","Type":"ContainerStarted","Data":"c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86"} Oct 30 14:03:07.580133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:07.580101 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:07.581642 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:07.581614 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6cf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:03:07.581773 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:07.581663 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:03:08.584215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:08.583587 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6cf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:03:08.584215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:08.583678 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:03:09.581381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:09.581339 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6cf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:03:09.581553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:09.581410 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:03:10.580644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:10.580606 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6cf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:03:10.581025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:10.580670 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:03:11.580745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:11.580648 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6cf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:03:11.580745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:11.580702 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:03:12.580810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:12.580772 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6cf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:03:12.581326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:12.580824 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:03:14.580532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:14.580487 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6cf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:03:14.580937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:14.580552 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:03:15.622325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:15.622290 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6cf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:03:15.622778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:15.622340 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:03:16.365697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:16.365661 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:16.366533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:16.366502 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:16.367374 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:16.367349 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:03:16.428856 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:16.428807 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podStartSLOduration=11.424703402 podStartE2EDuration="14.428791377s" totalImagesPullingTime="3.004087975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:06.803879054 +0000 UTC m=+816.604384500" watchObservedRunningTime="2026-10-30 14:03:16.428791377 +0000 UTC m=+826.229296825" Oct 30 14:03:17.311823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.311791 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg"] Oct 30 14:03:17.312204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.312179 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4c5f5033-acdc-4645-8d09-08da2c00e509" containerName="authorino" Oct 30 14:03:17.312204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.312203 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5f5033-acdc-4645-8d09-08da2c00e509" containerName="authorino" Oct 30 14:03:17.312284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.312275 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5f5033-acdc-4645-8d09-08da2c00e509" containerName="authorino" Oct 30 14:03:17.317795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.317772 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.350814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.350785 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg"] Oct 30 14:03:17.381812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.381785 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.381916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.381828 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.381916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.381862 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.381916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.381903 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.382038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.382025 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09180ae1-ace5-43a7-9b01-635b157cdf4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.382073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.382057 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.382105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.382081 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.382152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.382136 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.382222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.382182 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sjnw\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-kube-api-access-5sjnw\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483097 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483067 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483106 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483127 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483268 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sjnw\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-kube-api-access-5sjnw\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483372 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483410 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483447 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483471 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483504 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09180ae1-ace5-43a7-9b01-635b157cdf4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483579 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483765 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483745 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.483934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483891 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.484045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.483938 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.484660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.484633 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09180ae1-ace5-43a7-9b01-635b157cdf4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.485594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.485571 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.486264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.486244 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.509324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.509298 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.509498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.509477 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sjnw\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-kube-api-access-5sjnw\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hszvg\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.632606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.632540 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:17.952089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.952059 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg"] Oct 30 14:03:17.952365 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:03:17.952342 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09180ae1_ace5_43a7_9b01_635b157cdf4c.slice/crio-9eb1699be5e046645bd65445586f6e01af3e71893878e244779c8c60d4f5f48c WatchSource:0}: Error finding container 9eb1699be5e046645bd65445586f6e01af3e71893878e244779c8c60d4f5f48c: Status 404 returned error can't find the container with id 9eb1699be5e046645bd65445586f6e01af3e71893878e244779c8c60d4f5f48c Oct 30 14:03:17.955177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.955147 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:17.955282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.955232 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:17.955282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:17.955263 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:03:18.801674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:18.801634 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" event={"ID":"09180ae1-ace5-43a7-9b01-635b157cdf4c","Type":"ContainerStarted","Data":"674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28"} Oct 30 14:03:18.802067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:18.801684 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" event={"ID":"09180ae1-ace5-43a7-9b01-635b157cdf4c","Type":"ContainerStarted","Data":"9eb1699be5e046645bd65445586f6e01af3e71893878e244779c8c60d4f5f48c"} Oct 30 14:03:19.632820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:19.632786 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:19.634140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:19.634115 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:19.634259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:19.634165 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:03:20.632836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:20.632799 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:20.633226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:20.632852 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:03:21.633124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:21.633091 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:21.633522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:21.633139 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:03:22.633414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:22.633378 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:22.633896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:22.633439 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:03:23.633770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:23.633735 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:23.634165 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:23.633795 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:03:24.633470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:24.633432 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:03:24.633647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:24.633492 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:03:26.182202 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:26.182146 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:26.182679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:26.182472 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:26.183263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:26.183237 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:03:26.245298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:03:26.245257 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podStartSLOduration=9.245244519 podStartE2EDuration="9.245244519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:18.866559957 +0000 UTC m=+828.667065403" watchObservedRunningTime="2026-10-30 14:03:26.245244519 +0000 UTC m=+836.045750020" Oct 30 14:03:33.639446 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:03:33.639412 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:04:03.640305 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:04:03.640262 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:04:23.616518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.616482 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t"] Oct 30 14:04:23.620233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.620209 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.638022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.638002 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2df0ab-openshift-default-dockercfg-z8znz\"" Oct 30 14:04:23.652447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.652418 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t"] Oct 30 14:04:23.787322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787289 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.787506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787326 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.787506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787353 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad17e25-1d53-4cba-b2d4-bad295613862-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.787506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787468 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.787651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787532 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkh8\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-kube-api-access-5xkh8\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.787651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787595 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.787651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787631 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.787805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787730 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.787805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.787766 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.886731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.886658 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc"] Oct 30 14:04:23.888601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888573 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.888729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888616 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.888729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888675 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.888729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888717 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.888925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888756 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad17e25-1d53-4cba-b2d4-bad295613862-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.888925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888795 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.888925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888832 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkh8\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-kube-api-access-5xkh8\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.888925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888878 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.889124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.888940 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.889177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.889151 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.889260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.889146 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.889260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.889270 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.889562 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.889537 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.889774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.889750 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad17e25-1d53-4cba-b2d4-bad295613862-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.890603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.890584 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:23.891423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.891403 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.892011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.891987 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.916012 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.915986 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc"] Oct 30 14:04:23.919770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.919746 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkh8\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-kube-api-access-5xkh8\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.920683 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.920659 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.938135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.938112 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:23.970404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:23.970377 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-39529e-openshift-default-dockercfg-l4zq2\"" Oct 30 14:04:24.034607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.034579 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc"] Oct 30 14:04:24.038457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.038437 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.056418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.056396 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-da78c4-openshift-default-dockercfg-jkr4w\"" Oct 30 14:04:24.060874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.060845 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc"] Oct 30 14:04:24.090954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.090922 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.091087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.091040 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.091171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.091150 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.091248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.091179 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wpt\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-kube-api-access-c5wpt\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.091287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.091267 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.091320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.091292 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.091320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.091316 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.091416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.091398 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.091451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.091436 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.192415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192374 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.192621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192425 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.192621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192450 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.192621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192478 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.192621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192505 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.192621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192538 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.192621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192572 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.192621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192596 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.192621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192616 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192638 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192657 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7494b496-6844-484f-ab94-65830791c8a7-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192683 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192704 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192723 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqqt\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-kube-api-access-lzqqt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192752 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192767 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wpt\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-kube-api-access-c5wpt\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192783 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7494b496-6844-484f-ab94-65830791c8a7-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.193098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.192805 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.193643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.193173 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.193643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.193396 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.193759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.193642 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.194573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.194546 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.194831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.194806 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.195918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.195895 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.196116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.196097 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.220129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.220097 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.220636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.220617 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wpt\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-kube-api-access-c5wpt\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.221398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.221378 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:24.254911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.254881 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t"] Oct 30 14:04:24.255759 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:24.255729 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad17e25_1d53_4cba_b2d4_bad295613862.slice/crio-710444635af5dc4609f9f87c2095d17569b7695b4ddb0f349bc4f174a90a357d WatchSource:0}: Error finding container 710444635af5dc4609f9f87c2095d17569b7695b4ddb0f349bc4f174a90a357d: Status 404 returned error can't find the container with id 710444635af5dc4609f9f87c2095d17569b7695b4ddb0f349bc4f174a90a357d Oct 30 14:04:24.258099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.258057 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:24.258173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.258133 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:24.258173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.258168 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:24.294169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294134 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.294319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294253 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.294319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294291 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.294430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294319 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.294430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294357 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7494b496-6844-484f-ab94-65830791c8a7-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.294430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294397 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.294430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294427 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.294628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294455 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqqt\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-kube-api-access-lzqqt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.294628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.294505 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7494b496-6844-484f-ab94-65830791c8a7-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.295115 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.295091 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.295796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.295243 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7494b496-6844-484f-ab94-65830791c8a7-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.295796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.295486 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.295796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.295748 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.296365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.296338 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.297609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.297588 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7494b496-6844-484f-ab94-65830791c8a7-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.298076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.298059 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.321562 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.321537 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.321937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.321913 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqqt\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-kube-api-access-lzqqt\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.353988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.353954 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:24.540751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.540716 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc"] Oct 30 14:04:24.541876 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:24.541848 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6ba0ae_bf20_43b5_9dd0_d27882b9b7e0.slice/crio-b092f52f0b4b3271cc203d1957961f453bddee1c787315f74a830021203d27df WatchSource:0}: Error finding container b092f52f0b4b3271cc203d1957961f453bddee1c787315f74a830021203d27df: Status 404 returned error can't find the container with id b092f52f0b4b3271cc203d1957961f453bddee1c787315f74a830021203d27df Oct 30 14:04:24.544651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.544595 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:24.544754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.544679 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:24.544754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.544727 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:24.669509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.669480 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc"] Oct 30 14:04:24.671501 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:24.671476 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7494b496_6844_484f_ab94_65830791c8a7.slice/crio-99fbb69b5fab9cf10397fa34015df976394630c748f671dc1555ae200c9ea8c3 WatchSource:0}: Error finding container 99fbb69b5fab9cf10397fa34015df976394630c748f671dc1555ae200c9ea8c3: Status 404 returned error can't find the container with id 99fbb69b5fab9cf10397fa34015df976394630c748f671dc1555ae200c9ea8c3 Oct 30 14:04:24.675170 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.675135 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:24.675307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.675229 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:24.675307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:24.675282 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:25.115325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.115290 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" event={"ID":"7494b496-6844-484f-ab94-65830791c8a7","Type":"ContainerStarted","Data":"30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93"} Oct 30 14:04:25.115512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.115343 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" event={"ID":"7494b496-6844-484f-ab94-65830791c8a7","Type":"ContainerStarted","Data":"99fbb69b5fab9cf10397fa34015df976394630c748f671dc1555ae200c9ea8c3"} Oct 30 14:04:25.116896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.116868 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" event={"ID":"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0","Type":"ContainerStarted","Data":"44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a"} Oct 30 14:04:25.117035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.116918 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" event={"ID":"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0","Type":"ContainerStarted","Data":"b092f52f0b4b3271cc203d1957961f453bddee1c787315f74a830021203d27df"} Oct 30 14:04:25.118603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.118574 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" event={"ID":"9ad17e25-1d53-4cba-b2d4-bad295613862","Type":"ContainerStarted","Data":"912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3"} Oct 30 14:04:25.118699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.118622 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" event={"ID":"9ad17e25-1d53-4cba-b2d4-bad295613862","Type":"ContainerStarted","Data":"710444635af5dc4609f9f87c2095d17569b7695b4ddb0f349bc4f174a90a357d"} Oct 30 14:04:25.165129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.165082 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podStartSLOduration=1.165067219 podStartE2EDuration="1.165067219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:25.163521991 +0000 UTC m=+894.964027434" watchObservedRunningTime="2026-10-30 14:04:25.165067219 +0000 UTC m=+894.965572663" Oct 30 14:04:25.222161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.222130 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:25.226881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.226837 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podStartSLOduration=2.226823882 podStartE2EDuration="2.226823882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:25.226369066 +0000 UTC m=+895.026874511" watchObservedRunningTime="2026-10-30 14:04:25.226823882 +0000 UTC m=+895.027329326" Oct 30 14:04:25.227797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.227777 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:25.276475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.276421 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" podStartSLOduration=2.276406212 podStartE2EDuration="2.276406212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:25.275133911 +0000 UTC m=+895.075639357" watchObservedRunningTime="2026-10-30 14:04:25.276406212 +0000 UTC m=+895.076911656" Oct 30 14:04:25.354999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.354961 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:25.359938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.359914 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:25.938909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.938868 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:25.943944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:25.943917 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:26.123114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:26.123084 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:26.123313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:26.123145 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:26.123313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:26.123171 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:26.123861 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:26.123834 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:26.123993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:26.123971 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:26.124065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:26.124033 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:31.288995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.288961 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j"] Oct 30 14:04:31.294314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.294295 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:31.313539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.313517 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j"] Oct 30 14:04:31.325251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.325229 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-da78c4-dockercfg-6c2rb\"" Oct 30 14:04:31.325251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.325229 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-da78c4\"" Oct 30 14:04:31.369361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.369072 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09587b10-c0dd-49ca-be55-ba1ba847c2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j\" (UID: \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:31.369361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.369153 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmkvz\" (UniqueName: \"kubernetes.io/projected/09587b10-c0dd-49ca-be55-ba1ba847c2b9-kube-api-access-bmkvz\") pod \"payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j\" (UID: \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:31.376140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.376097 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt"] Oct 30 14:04:31.380361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.380333 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:31.399911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.399879 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt"] Oct 30 14:04:31.469889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.469854 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09587b10-c0dd-49ca-be55-ba1ba847c2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j\" (UID: \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:31.470062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.469912 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt\" (UID: \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:31.470062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.469942 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmkvz\" (UniqueName: \"kubernetes.io/projected/09587b10-c0dd-49ca-be55-ba1ba847c2b9-kube-api-access-bmkvz\") pod \"payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j\" (UID: \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:31.470215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.470117 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89q4\" (UniqueName: \"kubernetes.io/projected/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-kube-api-access-n89q4\") pod \"payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt\" (UID: \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:31.470647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.470622 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09587b10-c0dd-49ca-be55-ba1ba847c2b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j\" (UID: \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:31.495972 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.495945 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmkvz\" (UniqueName: \"kubernetes.io/projected/09587b10-c0dd-49ca-be55-ba1ba847c2b9-kube-api-access-bmkvz\") pod \"payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j\" (UID: \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:31.571145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.571090 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89q4\" (UniqueName: \"kubernetes.io/projected/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-kube-api-access-n89q4\") pod \"payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt\" (UID: \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:31.571265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.571144 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt\" (UID: \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:31.571830 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.571811 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt\" (UID: \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:31.597047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.597025 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89q4\" (UniqueName: \"kubernetes.io/projected/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-kube-api-access-n89q4\") pod \"payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt\" (UID: \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:31.607309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.607292 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:31.697521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.697493 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:31.905063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.905028 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j"] Oct 30 14:04:31.905698 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:31.905673 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09587b10_c0dd_49ca_be55_ba1ba847c2b9.slice/crio-db6ca62f671ae32a828ba0c1dd8b76aa1e9f6b3dffa74091e77bdf727f4fe7c3 WatchSource:0}: Error finding container db6ca62f671ae32a828ba0c1dd8b76aa1e9f6b3dffa74091e77bdf727f4fe7c3: Status 404 returned error can't find the container with id db6ca62f671ae32a828ba0c1dd8b76aa1e9f6b3dffa74091e77bdf727f4fe7c3 Oct 30 14:04:31.996078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:31.996045 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt"] Oct 30 14:04:31.996570 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:31.996540 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f5cdfb_5ad6_48a6_93c3_bbdeae520169.slice/crio-37045da4694d1b99469c9ff2473038b5cb8918b9a21babddf3406219f452663c WatchSource:0}: Error finding container 37045da4694d1b99469c9ff2473038b5cb8918b9a21babddf3406219f452663c: Status 404 returned error can't find the container with id 37045da4694d1b99469c9ff2473038b5cb8918b9a21babddf3406219f452663c Oct 30 14:04:32.121990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.121959 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z"] Oct 30 14:04:32.127066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.127047 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:32.144294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.144269 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z"] Oct 30 14:04:32.155552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.155482 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-39529e-dockercfg-7lzht\"" Oct 30 14:04:32.155552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.155492 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-39529e\"" Oct 30 14:04:32.156353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.156325 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" event={"ID":"09587b10-c0dd-49ca-be55-ba1ba847c2b9","Type":"ContainerStarted","Data":"db6ca62f671ae32a828ba0c1dd8b76aa1e9f6b3dffa74091e77bdf727f4fe7c3"} Oct 30 14:04:32.157683 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.157657 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" event={"ID":"79f5cdfb-5ad6-48a6-93c3-bbdeae520169","Type":"ContainerStarted","Data":"37045da4694d1b99469c9ff2473038b5cb8918b9a21babddf3406219f452663c"} Oct 30 14:04:32.191576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.190073 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f682f215-692e-4e8a-9ae2-76ea04c52ca3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z\" (UID: \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:32.191576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.190290 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnjh\" (UniqueName: \"kubernetes.io/projected/f682f215-692e-4e8a-9ae2-76ea04c52ca3-kube-api-access-wrnjh\") pod \"payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z\" (UID: \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:32.214678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.214646 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4"] Oct 30 14:04:32.219657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.219629 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:32.238773 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.238738 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4"] Oct 30 14:04:32.291460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.291420 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnjh\" (UniqueName: \"kubernetes.io/projected/f682f215-692e-4e8a-9ae2-76ea04c52ca3-kube-api-access-wrnjh\") pod \"payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z\" (UID: \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:32.291863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.291497 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f682f215-692e-4e8a-9ae2-76ea04c52ca3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z\" (UID: \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:32.291863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.291537 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fzl\" (UniqueName: \"kubernetes.io/projected/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-kube-api-access-r7fzl\") pod \"payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4\" (UID: \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:32.291863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.291581 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4\" (UID: \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:32.292168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.292148 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f682f215-692e-4e8a-9ae2-76ea04c52ca3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z\" (UID: \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:32.324319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.324289 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnjh\" (UniqueName: \"kubernetes.io/projected/f682f215-692e-4e8a-9ae2-76ea04c52ca3-kube-api-access-wrnjh\") pod \"payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z\" (UID: \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:32.392944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.392904 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4\" (UID: \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:32.393822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.393523 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fzl\" (UniqueName: \"kubernetes.io/projected/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-kube-api-access-r7fzl\") pod \"payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4\" (UID: \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:32.393822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.393768 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4\" (UID: \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:32.421261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.421170 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fzl\" (UniqueName: \"kubernetes.io/projected/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-kube-api-access-r7fzl\") pod \"payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4\" (UID: \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:32.440930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.440659 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:32.537574 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.537542 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:32.788133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.787582 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z"] Oct 30 14:04:32.804485 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:32.804333 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf682f215_692e_4e8a_9ae2_76ea04c52ca3.slice/crio-a8d168a7a240a13f719544774a31293824af7a948555b39088cc89e13d65f8c3 WatchSource:0}: Error finding container a8d168a7a240a13f719544774a31293824af7a948555b39088cc89e13d65f8c3: Status 404 returned error can't find the container with id a8d168a7a240a13f719544774a31293824af7a948555b39088cc89e13d65f8c3 Oct 30 14:04:32.862571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:32.861966 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4"] Oct 30 14:04:32.867365 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:32.867321 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cac6cc1_b6eb_4ef5_adf8_9c0eac9f393b.slice/crio-77f46aecf87382f11bc6dc21fcf436ab8ef2faefa7e9e7684be0ce1fe67a8d5a WatchSource:0}: Error finding container 77f46aecf87382f11bc6dc21fcf436ab8ef2faefa7e9e7684be0ce1fe67a8d5a: Status 404 returned error can't find the container with id 77f46aecf87382f11bc6dc21fcf436ab8ef2faefa7e9e7684be0ce1fe67a8d5a Oct 30 14:04:33.128406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.128292 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49"] Oct 30 14:04:33.132790 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.132739 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:33.139244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.139156 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49"] Oct 30 14:04:33.158348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.158318 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2df0ab-dockercfg-994gs\"" Oct 30 14:04:33.159678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.159638 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2df0ab\"" Oct 30 14:04:33.165927 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.165896 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" event={"ID":"f682f215-692e-4e8a-9ae2-76ea04c52ca3","Type":"ContainerStarted","Data":"a8d168a7a240a13f719544774a31293824af7a948555b39088cc89e13d65f8c3"} Oct 30 14:04:33.168018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.167962 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" event={"ID":"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b","Type":"ContainerStarted","Data":"77f46aecf87382f11bc6dc21fcf436ab8ef2faefa7e9e7684be0ce1fe67a8d5a"} Oct 30 14:04:33.226944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.226910 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650319fd-aeda-4ba4-9594-51ef46a181a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49\" (UID: \"650319fd-aeda-4ba4-9594-51ef46a181a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:33.227158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.226962 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhqj\" (UniqueName: \"kubernetes.io/projected/650319fd-aeda-4ba4-9594-51ef46a181a9-kube-api-access-rhhqj\") pod \"payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49\" (UID: \"650319fd-aeda-4ba4-9594-51ef46a181a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:33.327726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.327686 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650319fd-aeda-4ba4-9594-51ef46a181a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49\" (UID: \"650319fd-aeda-4ba4-9594-51ef46a181a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:33.328250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.327758 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhqj\" (UniqueName: \"kubernetes.io/projected/650319fd-aeda-4ba4-9594-51ef46a181a9-kube-api-access-rhhqj\") pod \"payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49\" (UID: \"650319fd-aeda-4ba4-9594-51ef46a181a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:33.329005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.328938 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650319fd-aeda-4ba4-9594-51ef46a181a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49\" (UID: \"650319fd-aeda-4ba4-9594-51ef46a181a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:33.361803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.361740 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhqj\" (UniqueName: \"kubernetes.io/projected/650319fd-aeda-4ba4-9594-51ef46a181a9-kube-api-access-rhhqj\") pod \"payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49\" (UID: \"650319fd-aeda-4ba4-9594-51ef46a181a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:33.459573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.459322 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:33.657570 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:04:33.657529 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:04:33.792229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:33.792135 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49"] Oct 30 14:04:34.176479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:34.176440 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" event={"ID":"650319fd-aeda-4ba4-9594-51ef46a181a9","Type":"ContainerStarted","Data":"ca8f346ff9c08b5ebce5d7180a7c221bec4f7fd8896edcd35cc38299a3128cc7"} Oct 30 14:04:35.406489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.404167 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9"] Oct 30 14:04:35.409154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.409126 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:35.425857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.425783 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9"] Oct 30 14:04:35.437684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.437364 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-a7bb0c-dockercfg-v7pzz\"" Oct 30 14:04:35.519171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.519146 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-a7bb0c\"" Oct 30 14:04:35.562991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.562954 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mhb\" (UniqueName: \"kubernetes.io/projected/137c45cd-711f-46e6-a808-7e396102fb7f-kube-api-access-t2mhb\") pod \"payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9\" (UID: \"137c45cd-711f-46e6-a808-7e396102fb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:35.563094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.563066 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c45cd-711f-46e6-a808-7e396102fb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9\" (UID: \"137c45cd-711f-46e6-a808-7e396102fb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:35.664326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.664057 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mhb\" (UniqueName: \"kubernetes.io/projected/137c45cd-711f-46e6-a808-7e396102fb7f-kube-api-access-t2mhb\") pod \"payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9\" (UID: \"137c45cd-711f-46e6-a808-7e396102fb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:35.664326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.664167 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c45cd-711f-46e6-a808-7e396102fb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9\" (UID: \"137c45cd-711f-46e6-a808-7e396102fb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:35.665050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.665022 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c45cd-711f-46e6-a808-7e396102fb7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9\" (UID: \"137c45cd-711f-46e6-a808-7e396102fb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:35.692494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.691774 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mhb\" (UniqueName: \"kubernetes.io/projected/137c45cd-711f-46e6-a808-7e396102fb7f-kube-api-access-t2mhb\") pod \"payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9\" (UID: \"137c45cd-711f-46e6-a808-7e396102fb7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:35.728120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:35.728094 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:36.047179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.047151 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9"] Oct 30 14:04:36.048536 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:36.048510 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137c45cd_711f_46e6_a808_7e396102fb7f.slice/crio-78e8764e71b7954af6bb89ddb473399a4444b6bb0a55e05aa8ec2d2fa45427f6 WatchSource:0}: Error finding container 78e8764e71b7954af6bb89ddb473399a4444b6bb0a55e05aa8ec2d2fa45427f6: Status 404 returned error can't find the container with id 78e8764e71b7954af6bb89ddb473399a4444b6bb0a55e05aa8ec2d2fa45427f6 Oct 30 14:04:36.195379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.195328 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" event={"ID":"09587b10-c0dd-49ca-be55-ba1ba847c2b9","Type":"ContainerStarted","Data":"df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc"} Oct 30 14:04:36.195533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.195505 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:36.196913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.196886 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" event={"ID":"f682f215-692e-4e8a-9ae2-76ea04c52ca3","Type":"ContainerStarted","Data":"e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f"} Oct 30 14:04:36.197041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.197022 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:36.198769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.198739 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" event={"ID":"137c45cd-711f-46e6-a808-7e396102fb7f","Type":"ContainerStarted","Data":"ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f"} Oct 30 14:04:36.198881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.198785 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" event={"ID":"137c45cd-711f-46e6-a808-7e396102fb7f","Type":"ContainerStarted","Data":"78e8764e71b7954af6bb89ddb473399a4444b6bb0a55e05aa8ec2d2fa45427f6"} Oct 30 14:04:36.198881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.198867 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:36.200265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.200241 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" event={"ID":"79f5cdfb-5ad6-48a6-93c3-bbdeae520169","Type":"ContainerStarted","Data":"a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77"} Oct 30 14:04:36.200384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.200358 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:36.201885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.201859 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" event={"ID":"650319fd-aeda-4ba4-9594-51ef46a181a9","Type":"ContainerStarted","Data":"a87132cdbc0e64649aebf220ba321cfdb637e8976020fb3780651545b426ff68"} Oct 30 14:04:36.201984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.201916 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:36.203646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.203622 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" event={"ID":"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b","Type":"ContainerStarted","Data":"3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d"} Oct 30 14:04:36.203780 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.203763 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:36.244785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.244722 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" podStartSLOduration=1.732876039 podStartE2EDuration="5.244702786s" totalImagesPullingTime="3.511826747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:36.243462806 +0000 UTC m=+906.043968273" watchObservedRunningTime="2026-10-30 14:04:36.244702786 +0000 UTC m=+906.045208232" Oct 30 14:04:36.288080 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.287974 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" podStartSLOduration=1.853060024 podStartE2EDuration="5.287956894s" totalImagesPullingTime="3.43489687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:36.287289351 +0000 UTC m=+906.087794794" watchObservedRunningTime="2026-10-30 14:04:36.287956894 +0000 UTC m=+906.088462333" Oct 30 14:04:36.332912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.332868 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" podStartSLOduration=1.083056434 podStartE2EDuration="4.332855444s" totalImagesPullingTime="3.24979901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:36.331119571 +0000 UTC m=+906.131625016" watchObservedRunningTime="2026-10-30 14:04:36.332855444 +0000 UTC m=+906.133360888" Oct 30 14:04:36.376540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.376490 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" podStartSLOduration=1.376478427 podStartE2EDuration="1.376478427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:36.374721909 +0000 UTC m=+906.175227367" watchObservedRunningTime="2026-10-30 14:04:36.376478427 +0000 UTC m=+906.176983871" Oct 30 14:04:36.419525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.419478 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" podStartSLOduration=1.548843949 podStartE2EDuration="3.419465968s" totalImagesPullingTime="1.870622019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:36.418357816 +0000 UTC m=+906.218863262" watchObservedRunningTime="2026-10-30 14:04:36.419465968 +0000 UTC m=+906.219971416" Oct 30 14:04:36.466047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:36.465993 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" podStartSLOduration=1.3335458629999999 podStartE2EDuration="4.465977786s" totalImagesPullingTime="3.132431923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:36.464927482 +0000 UTC m=+906.265432926" watchObservedRunningTime="2026-10-30 14:04:36.465977786 +0000 UTC m=+906.266483268" Oct 30 14:04:37.001730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.001691 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn"] Oct 30 14:04:37.005944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.005920 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:37.023282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.023256 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn"] Oct 30 14:04:37.034633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.034607 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-46b13b-dockercfg-7kw4z\"" Oct 30 14:04:37.072290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.072242 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-46b13b\"" Oct 30 14:04:37.096216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.084401 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81282091-cc28-4f1d-b6bd-aa68cadd63c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn\" (UID: \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:37.096216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.084454 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxlf\" (UniqueName: \"kubernetes.io/projected/81282091-cc28-4f1d-b6bd-aa68cadd63c0-kube-api-access-cqxlf\") pod \"payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn\" (UID: \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:37.193124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.191878 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81282091-cc28-4f1d-b6bd-aa68cadd63c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn\" (UID: \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:37.193124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.191934 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxlf\" (UniqueName: \"kubernetes.io/projected/81282091-cc28-4f1d-b6bd-aa68cadd63c0-kube-api-access-cqxlf\") pod \"payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn\" (UID: \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:37.193124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.193070 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81282091-cc28-4f1d-b6bd-aa68cadd63c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn\" (UID: \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:37.226921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.226883 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxlf\" (UniqueName: \"kubernetes.io/projected/81282091-cc28-4f1d-b6bd-aa68cadd63c0-kube-api-access-cqxlf\") pod \"payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn\" (UID: \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:37.319702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.319604 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:37.634084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:37.634055 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn"] Oct 30 14:04:37.635559 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:37.635535 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81282091_cc28_4f1d_b6bd_aa68cadd63c0.slice/crio-9aaf6ac8f33658d787bd5830fc80ae3273313d8e507d3b2dc0f78d7fd055487f WatchSource:0}: Error finding container 9aaf6ac8f33658d787bd5830fc80ae3273313d8e507d3b2dc0f78d7fd055487f: Status 404 returned error can't find the container with id 9aaf6ac8f33658d787bd5830fc80ae3273313d8e507d3b2dc0f78d7fd055487f Oct 30 14:04:38.236794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:38.236746 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" event={"ID":"81282091-cc28-4f1d-b6bd-aa68cadd63c0","Type":"ContainerStarted","Data":"b43edd5ecf080e5a0b9b98f478e9592175e0ea2dba14966a05ba3e0204af12da"} Oct 30 14:04:38.236977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:38.236812 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" event={"ID":"81282091-cc28-4f1d-b6bd-aa68cadd63c0","Type":"ContainerStarted","Data":"9aaf6ac8f33658d787bd5830fc80ae3273313d8e507d3b2dc0f78d7fd055487f"} Oct 30 14:04:38.236977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:38.236947 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:38.281464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:38.281404 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" podStartSLOduration=2.281390316 podStartE2EDuration="2.281390316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:38.279970915 +0000 UTC m=+908.080476358" watchObservedRunningTime="2026-10-30 14:04:38.281390316 +0000 UTC m=+908.081895760" Oct 30 14:04:42.961273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:42.961068 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g"] Oct 30 14:04:42.969213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:42.966745 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.003214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.002328 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:04:43.003214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.002436 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:04:43.032251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.030911 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g"] Oct 30 14:04:43.075214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.063930 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-54whp\"" Oct 30 14:04:43.075214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.064116 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 14:04:43.075214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.068249 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-a7bb0c-serving-cert\"" Oct 30 14:04:43.129585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.122324 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c9278f-54d3-49cb-975c-f49372a9fd1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.129585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.122424 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.129585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.122465 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-metrics-certs\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.129585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.122585 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjvr9\" (UniqueName: \"kubernetes.io/projected/60c9278f-54d3-49cb-975c-f49372a9fd1c-kube-api-access-kjvr9\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.225473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.224180 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvr9\" (UniqueName: \"kubernetes.io/projected/60c9278f-54d3-49cb-975c-f49372a9fd1c-kube-api-access-kjvr9\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.225473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.224335 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c9278f-54d3-49cb-975c-f49372a9fd1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.225473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.224413 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.225473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.224461 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-metrics-certs\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.225473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.225349 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c9278f-54d3-49cb-975c-f49372a9fd1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.227716 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.227687 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-metrics-certs\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.227828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.227770 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.257214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.253860 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvr9\" (UniqueName: \"kubernetes.io/projected/60c9278f-54d3-49cb-975c-f49372a9fd1c-kube-api-access-kjvr9\") pod \"maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.433053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.433020 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:43.909537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:43.899362 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g"] Oct 30 14:04:43.920219 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:43.919501 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c9278f_54d3_49cb_975c_f49372a9fd1c.slice/crio-81ea3e8c48b7d5aa456a17a9f0ec22abd2a6bd0b85e188b66a10f91ce7cbc53d WatchSource:0}: Error finding container 81ea3e8c48b7d5aa456a17a9f0ec22abd2a6bd0b85e188b66a10f91ce7cbc53d: Status 404 returned error can't find the container with id 81ea3e8c48b7d5aa456a17a9f0ec22abd2a6bd0b85e188b66a10f91ce7cbc53d Oct 30 14:04:44.324092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:44.324055 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" event={"ID":"60c9278f-54d3-49cb-975c-f49372a9fd1c","Type":"ContainerStarted","Data":"81ea3e8c48b7d5aa456a17a9f0ec22abd2a6bd0b85e188b66a10f91ce7cbc53d"} Oct 30 14:04:47.239087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:47.235537 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:04:47.239087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:47.237463 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:04:47.239640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:47.239162 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:04:47.239640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:47.239245 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:04:47.239755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:47.239653 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:04:47.239800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:47.239759 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:04:48.056513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.056421 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz"] Oct 30 14:04:48.064371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.064344 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz"] Oct 30 14:04:48.079067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.073403 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:48.098277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.098238 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-ca229c-dockercfg-2x6zf\"" Oct 30 14:04:48.098408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.098380 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-ca229c\"" Oct 30 14:04:48.108353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.108320 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf"] Oct 30 14:04:48.148248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.147956 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22x6\" (UniqueName: \"kubernetes.io/projected/4b25e4f7-687c-429f-9395-951ebe00e027-kube-api-access-w22x6\") pod \"payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz\" (UID: \"4b25e4f7-687c-429f-9395-951ebe00e027\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:48.148248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.148008 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25e4f7-687c-429f-9395-951ebe00e027-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz\" (UID: \"4b25e4f7-687c-429f-9395-951ebe00e027\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:48.178446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.178412 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf"] Oct 30 14:04:48.178617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.178517 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:48.250210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.249256 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592a7ead-f353-4b52-9fb2-f2bf41af4119-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf\" (UID: \"592a7ead-f353-4b52-9fb2-f2bf41af4119\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:48.250210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.249328 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22x6\" (UniqueName: \"kubernetes.io/projected/4b25e4f7-687c-429f-9395-951ebe00e027-kube-api-access-w22x6\") pod \"payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz\" (UID: \"4b25e4f7-687c-429f-9395-951ebe00e027\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:48.250210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.249367 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25e4f7-687c-429f-9395-951ebe00e027-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz\" (UID: \"4b25e4f7-687c-429f-9395-951ebe00e027\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:48.250210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.249394 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqxx\" (UniqueName: \"kubernetes.io/projected/592a7ead-f353-4b52-9fb2-f2bf41af4119-kube-api-access-xsqxx\") pod \"payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf\" (UID: \"592a7ead-f353-4b52-9fb2-f2bf41af4119\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:48.250825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.250593 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25e4f7-687c-429f-9395-951ebe00e027-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz\" (UID: \"4b25e4f7-687c-429f-9395-951ebe00e027\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:48.278294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.278159 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22x6\" (UniqueName: \"kubernetes.io/projected/4b25e4f7-687c-429f-9395-951ebe00e027-kube-api-access-w22x6\") pod \"payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz\" (UID: \"4b25e4f7-687c-429f-9395-951ebe00e027\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:48.352465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.351173 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592a7ead-f353-4b52-9fb2-f2bf41af4119-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf\" (UID: \"592a7ead-f353-4b52-9fb2-f2bf41af4119\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:48.352465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.351270 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqxx\" (UniqueName: \"kubernetes.io/projected/592a7ead-f353-4b52-9fb2-f2bf41af4119-kube-api-access-xsqxx\") pod \"payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf\" (UID: \"592a7ead-f353-4b52-9fb2-f2bf41af4119\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:48.352465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.352390 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592a7ead-f353-4b52-9fb2-f2bf41af4119-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf\" (UID: \"592a7ead-f353-4b52-9fb2-f2bf41af4119\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:48.383960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.383902 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqxx\" (UniqueName: \"kubernetes.io/projected/592a7ead-f353-4b52-9fb2-f2bf41af4119-kube-api-access-xsqxx\") pod \"payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf\" (UID: \"592a7ead-f353-4b52-9fb2-f2bf41af4119\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:48.470468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.470428 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:48.500212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.500149 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:48.844592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.844045 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf"] Oct 30 14:04:48.847426 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:48.847290 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592a7ead_f353_4b52_9fb2_f2bf41af4119.slice/crio-d62d6c9657f23463c8e5f032eaf74c6b6e76e2e580631e936a80e0f927adb78b WatchSource:0}: Error finding container d62d6c9657f23463c8e5f032eaf74c6b6e76e2e580631e936a80e0f927adb78b: Status 404 returned error can't find the container with id d62d6c9657f23463c8e5f032eaf74c6b6e76e2e580631e936a80e0f927adb78b Oct 30 14:04:48.864636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:48.864608 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz"] Oct 30 14:04:48.872126 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:48.872076 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b25e4f7_687c_429f_9395_951ebe00e027.slice/crio-927597a4575a2b7433c5de8f2e110de3a4a6e476cd70d093bdfbbabed2800870 WatchSource:0}: Error finding container 927597a4575a2b7433c5de8f2e110de3a4a6e476cd70d093bdfbbabed2800870: Status 404 returned error can't find the container with id 927597a4575a2b7433c5de8f2e110de3a4a6e476cd70d093bdfbbabed2800870 Oct 30 14:04:49.246229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.245414 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:04:49.355489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.355448 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" event={"ID":"60c9278f-54d3-49cb-975c-f49372a9fd1c","Type":"ContainerStarted","Data":"e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25"} Oct 30 14:04:49.356670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.356394 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:49.358910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.358430 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" event={"ID":"4b25e4f7-687c-429f-9395-951ebe00e027","Type":"ContainerStarted","Data":"58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c"} Oct 30 14:04:49.358910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.358475 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" event={"ID":"4b25e4f7-687c-429f-9395-951ebe00e027","Type":"ContainerStarted","Data":"927597a4575a2b7433c5de8f2e110de3a4a6e476cd70d093bdfbbabed2800870"} Oct 30 14:04:49.363004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.359055 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:04:49.363004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.361384 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" event={"ID":"592a7ead-f353-4b52-9fb2-f2bf41af4119","Type":"ContainerStarted","Data":"22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c"} Oct 30 14:04:49.363004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.361437 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" event={"ID":"592a7ead-f353-4b52-9fb2-f2bf41af4119","Type":"ContainerStarted","Data":"d62d6c9657f23463c8e5f032eaf74c6b6e76e2e580631e936a80e0f927adb78b"} Oct 30 14:04:49.363004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.362057 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:04:49.406738 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.406672 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" podStartSLOduration=2.618570982 podStartE2EDuration="7.406653687s" totalImagesPullingTime="4.788082705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:49.406232394 +0000 UTC m=+919.206737835" watchObservedRunningTime="2026-10-30 14:04:49.406653687 +0000 UTC m=+919.207159133" Oct 30 14:04:49.454523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.454458 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" podStartSLOduration=1.4544394 podStartE2EDuration="1.4544394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:49.45306506 +0000 UTC m=+919.253570506" watchObservedRunningTime="2026-10-30 14:04:49.4544394 +0000 UTC m=+919.254944844" Oct 30 14:04:49.499261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:49.498599 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" podStartSLOduration=2.49857931 podStartE2EDuration="2.49857931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:49.497787059 +0000 UTC m=+919.298292503" watchObservedRunningTime="2026-10-30 14:04:49.49857931 +0000 UTC m=+919.299084758" Oct 30 14:04:54.050694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.050634 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn"] Oct 30 14:04:54.053574 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.053559 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.077702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.077676 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn"] Oct 30 14:04:54.118885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.118850 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knjrk\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-kube-api-access-knjrk\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.119013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.118901 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.119013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.118995 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.119116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.119024 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.119116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.119045 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.119222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.119175 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.119284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.119259 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.119346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.119293 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.119346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.119331 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.220879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.220841 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knjrk\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-kube-api-access-knjrk\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.220892 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.220942 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.220973 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221109 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221180 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221323 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221256 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221323 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221276 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221332 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221350 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221560 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221610 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221673 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.221737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.221706 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.223501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.223476 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.223697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.223680 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.246888 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.246857 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.247011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.246996 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knjrk\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-kube-api-access-knjrk\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.368946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.368878 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:54.395837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.395803 5824 generic.go:356] "Generic (PLEG): container finished" podID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerID="912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3" exitCode=0 Oct 30 14:04:54.395985 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.395878 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" event={"ID":"9ad17e25-1d53-4cba-b2d4-bad295613862","Type":"ContainerDied","Data":"912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3"} Oct 30 14:04:54.396460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.396420 5824 scope.go:122] "RemoveContainer" containerID="912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3" Oct 30 14:04:54.397868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.397844 5824 generic.go:356] "Generic (PLEG): container finished" podID="7494b496-6844-484f-ab94-65830791c8a7" containerID="30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93" exitCode=0 Oct 30 14:04:54.397990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.397918 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" event={"ID":"7494b496-6844-484f-ab94-65830791c8a7","Type":"ContainerDied","Data":"30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93"} Oct 30 14:04:54.398059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.398007 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.398120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.398076 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.398179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.398117 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.398350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.398333 5824 scope.go:122] "RemoveContainer" containerID="30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93" Oct 30 14:04:54.399682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.399650 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.399782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.399734 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.399848 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.399786 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.399905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.399869 5824 generic.go:356] "Generic (PLEG): container finished" podID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerID="44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a" exitCode=0 Oct 30 14:04:54.399964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.399933 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" event={"ID":"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0","Type":"ContainerDied","Data":"44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a"} Oct 30 14:04:54.400296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.400283 5824 scope.go:122] "RemoveContainer" containerID="44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a" Oct 30 14:04:54.401574 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.401543 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.401655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.401617 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.401690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.401661 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.744515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.743530 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn"] Oct 30 14:04:54.744713 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:04:54.744676 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ece2f2_3dfd_4b6f_b36b_b1f8cbfd97e9.slice/crio-258d07766d872dbc826f6fc221c0e9dd2114810c63edce002ea8f0fe1cca0e43 WatchSource:0}: Error finding container 258d07766d872dbc826f6fc221c0e9dd2114810c63edce002ea8f0fe1cca0e43: Status 404 returned error can't find the container with id 258d07766d872dbc826f6fc221c0e9dd2114810c63edce002ea8f0fe1cca0e43 Oct 30 14:04:54.747578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.747543 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.747697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.747616 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:54.747697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:54.747687 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:04:55.494213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.494031 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" event={"ID":"7494b496-6844-484f-ab94-65830791c8a7","Type":"ContainerStarted","Data":"25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114"} Oct 30 14:04:55.509215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.506824 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" event={"ID":"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0","Type":"ContainerStarted","Data":"b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4"} Oct 30 14:04:55.535979 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.535930 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" event={"ID":"9ad17e25-1d53-4cba-b2d4-bad295613862","Type":"ContainerStarted","Data":"761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3"} Oct 30 14:04:55.538316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.538281 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" event={"ID":"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9","Type":"ContainerStarted","Data":"a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b"} Oct 30 14:04:55.540590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.540555 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" event={"ID":"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9","Type":"ContainerStarted","Data":"258d07766d872dbc826f6fc221c0e9dd2114810c63edce002ea8f0fe1cca0e43"} Oct 30 14:04:55.614213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.613949 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podStartSLOduration=1.613929405 podStartE2EDuration="1.613929405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:04:55.613520198 +0000 UTC m=+925.414025643" watchObservedRunningTime="2026-10-30 14:04:55.613929405 +0000 UTC m=+925.414434851" Oct 30 14:04:55.948212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.940796 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:55.948212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.941021 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:04:55.948212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:55.941074 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:04:56.127214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.126343 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:56.127214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.126485 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:56.127214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.126569 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:04:56.240273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.225556 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:04:56.240273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.225740 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 14:04:56.240273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.225784 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 14:04:56.376261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.365298 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 14:04:56.376261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.365359 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 14:04:56.376261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.367973 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:04:56.376261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.375899 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:04:56.376261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.376015 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 14:04:56.376261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.376061 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 14:04:56.423434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.423399 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:04:56.942215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.938630 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:04:56.942215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:56.938692 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:04:57.226219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:57.225656 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 14:04:57.226219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:57.225727 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 14:04:57.358216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:57.357607 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 14:04:57.358216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:57.357672 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 14:04:57.392221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:57.390141 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 14:04:57.392221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:57.390221 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 14:04:57.939597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:57.939552 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:04:57.939902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:57.939868 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:04:58.223713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:58.223621 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 14:04:58.224310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:58.224275 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 14:04:58.359214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:58.358559 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 14:04:58.359214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:58.358618 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 14:04:58.381216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:58.369630 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 14:04:58.381216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:58.369679 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 14:04:58.938497 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:58.938458 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:04:58.944343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:58.944304 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:04:59.238670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:59.238563 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 14:04:59.238670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:59.238629 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 14:04:59.356234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:59.355692 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 14:04:59.356234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:59.355769 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 14:04:59.370583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:59.370537 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 14:04:59.370866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:59.370831 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 14:04:59.942215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:59.941661 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:04:59.942215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:04:59.941733 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:05:00.227283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:00.226618 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 14:05:00.227283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:00.226685 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 14:05:00.362339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:00.362303 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 14:05:00.362897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:00.362870 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 14:05:00.370577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:00.370543 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 14:05:00.370839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:00.370812 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 14:05:00.946215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:00.945608 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:05:00.946215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:00.945674 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:05:01.225273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:01.225144 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 14:05:01.225273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:01.225242 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 14:05:01.355215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:01.354586 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 14:05:01.355215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:01.354654 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 14:05:01.377217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:01.374624 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 14:05:01.377217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:01.374684 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 14:05:01.393490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:01.392501 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:05:01.394592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:01.394478 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:05:02.024242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.024208 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:05:02.025976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.025952 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:05:02.228324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.228291 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:05:02.229753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.229728 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:05:02.363079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.363004 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:05:02.365613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.365584 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:05:02.376854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.376827 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:05:02.377136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.377115 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:05:02.378744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:02.378701 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:05:03.302770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.302737 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md"] Oct 30 14:05:03.306868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.306849 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.328988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.328960 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md"] Oct 30 14:05:03.338004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.337984 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-ca229c-serving-cert\"" Oct 30 14:05:03.362160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.362136 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/02489a04-3f89-490e-a290-5608d5c61dd6-kube-api-access-nj5ps\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.362311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.362268 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02489a04-3f89-490e-a290-5608d5c61dd6-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.362383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.362315 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.362469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.362451 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-metrics-certs\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.463692 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.463659 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-metrics-certs\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.464116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.463790 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/02489a04-3f89-490e-a290-5608d5c61dd6-kube-api-access-nj5ps\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.464116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.463867 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02489a04-3f89-490e-a290-5608d5c61dd6-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.464116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.463903 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.464116 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:05:03.464090 5824 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-serving-cert: secret "maas-api-e2e-worker-w1-ca229c-serving-cert" not found Oct 30 14:05:03.464290 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:05:03.464153 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls podName:02489a04-3f89-490e-a290-5608d5c61dd6 nodeName:}" failed. No retries permitted until 2026-10-30 14:05:03.964139065 +0000 UTC m=+933.764644487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls") pod "maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" (UID: "02489a04-3f89-490e-a290-5608d5c61dd6") : secret "maas-api-e2e-worker-w1-ca229c-serving-cert" not found Oct 30 14:05:03.464516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.464497 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02489a04-3f89-490e-a290-5608d5c61dd6-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.465996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.465980 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-metrics-certs\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.489175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.489149 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/02489a04-3f89-490e-a290-5608d5c61dd6-kube-api-access-nj5ps\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.640001 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:05:03.639920 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:05:03.968768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.968733 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:03.971115 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:03.971088 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:04.219123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:04.219031 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:04.521090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:04.521060 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md"] Oct 30 14:05:04.521500 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:05:04.521477 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02489a04_3f89_490e_a290_5608d5c61dd6.slice/crio-7b278435d81af5c21c12a820e8514a44ca10d1117cad3c859693dea768aa5822 WatchSource:0}: Error finding container 7b278435d81af5c21c12a820e8514a44ca10d1117cad3c859693dea768aa5822: Status 404 returned error can't find the container with id 7b278435d81af5c21c12a820e8514a44ca10d1117cad3c859693dea768aa5822 Oct 30 14:05:04.522906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:04.522887 5824 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:05:04.758101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:04.758063 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" event={"ID":"02489a04-3f89-490e-a290-5608d5c61dd6","Type":"ContainerStarted","Data":"7b278435d81af5c21c12a820e8514a44ca10d1117cad3c859693dea768aa5822"} Oct 30 14:05:05.765022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:05.764989 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" event={"ID":"02489a04-3f89-490e-a290-5608d5c61dd6","Type":"ContainerStarted","Data":"aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0"} Oct 30 14:05:05.765435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:05.765095 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:05.819604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:05.819540 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" podStartSLOduration=2.415785317 podStartE2EDuration="2.819523992s" totalImagesPullingTime="403.738675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:05.810430663 +0000 UTC m=+935.610936107" watchObservedRunningTime="2026-10-30 14:05:05.819523992 +0000 UTC m=+935.620029437" Oct 30 14:05:11.777931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:11.777901 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:05:23.730222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.730161 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s"] Oct 30 14:05:23.734253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.734228 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.814707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.814663 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-00926c-openshift-default-dockercfg-2jkx6\"" Oct 30 14:05:23.817267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.817239 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s"] Oct 30 14:05:23.917845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.917809 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.917845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.917847 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.918087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.917869 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.918087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.917924 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.918230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.918136 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.918277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.918252 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.918331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.918274 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.918331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.918318 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:23.918443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:23.918405 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbb4z\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-kube-api-access-sbb4z\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.019821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.019735 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.019821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.019800 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.019827 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.019864 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.019989 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbb4z\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-kube-api-access-sbb4z\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020095 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020120 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020140 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020168 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020279 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020351 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020567 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020662 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.020741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.020704 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.022126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.022106 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.022546 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.022524 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.049057 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.049034 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.049332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.049313 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbb4z\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-kube-api-access-sbb4z\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.116318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.116295 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:24.428870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.428847 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s"] Oct 30 14:05:24.430096 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:05:24.430065 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e55f6cb_aeb1_4628_af45_1677d1734c8b.slice/crio-6d1079adb8726f117301e4543d84b6c1812b99cfc45f757b1b0754e9461a463a WatchSource:0}: Error finding container 6d1079adb8726f117301e4543d84b6c1812b99cfc45f757b1b0754e9461a463a: Status 404 returned error can't find the container with id 6d1079adb8726f117301e4543d84b6c1812b99cfc45f757b1b0754e9461a463a Oct 30 14:05:24.432632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.432589 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:24.432711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.432682 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:24.432748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.432724 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:24.848412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.848300 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx"] Oct 30 14:05:24.852343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.852318 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:24.879634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.879555 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx"] Oct 30 14:05:24.880256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.880173 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-46b13b\"/\"openshift-service-ca.crt\"" Oct 30 14:05:24.880838 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.880811 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-46b13b\"/\"simulator-w4-46b13b-kserve-self-signed-certs\"" Oct 30 14:05:24.881063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.881045 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-46b13b\"/\"kube-root-ca.crt\"" Oct 30 14:05:24.881169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.880283 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-46b13b\"/\"default-dockercfg-r9dsj\"" Oct 30 14:05:24.894615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.894576 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" event={"ID":"1e55f6cb-aeb1-4628-af45-1677d1734c8b","Type":"ContainerStarted","Data":"5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a"} Oct 30 14:05:24.894796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.894644 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" event={"ID":"1e55f6cb-aeb1-4628-af45-1677d1734c8b","Type":"ContainerStarted","Data":"6d1079adb8726f117301e4543d84b6c1812b99cfc45f757b1b0754e9461a463a"} Oct 30 14:05:24.971247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:24.968018 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podStartSLOduration=1.967997446 podStartE2EDuration="1.967997446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:24.967670749 +0000 UTC m=+954.768176207" watchObservedRunningTime="2026-10-30 14:05:24.967997446 +0000 UTC m=+954.768502892" Oct 30 14:05:25.032968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.032935 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-home\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.033109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.032975 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-tmp-dir\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.033109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.033023 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsgkk\" (UniqueName: \"kubernetes.io/projected/057f7e4f-090a-42ab-92a1-cc9359882375-kube-api-access-qsgkk\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.033378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.033360 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-dshm\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.033436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.033421 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-model-cache\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.033521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.033502 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057f7e4f-090a-42ab-92a1-cc9359882375-tls-certs\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.116883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.116793 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:25.118239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.118180 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:05:25.118378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.118263 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 14:05:25.135282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135259 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-model-cache\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.135407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135290 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057f7e4f-090a-42ab-92a1-cc9359882375-tls-certs\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.135407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135378 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-home\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.135407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135400 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-tmp-dir\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.135531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135428 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsgkk\" (UniqueName: \"kubernetes.io/projected/057f7e4f-090a-42ab-92a1-cc9359882375-kube-api-access-qsgkk\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.135531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135492 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-dshm\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.135745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135720 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-model-cache\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.135813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135752 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-home\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.135813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.135794 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-tmp-dir\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.137589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.137567 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-dshm\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.137862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.137841 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057f7e4f-090a-42ab-92a1-cc9359882375-tls-certs\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.173213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.172321 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsgkk\" (UniqueName: \"kubernetes.io/projected/057f7e4f-090a-42ab-92a1-cc9359882375-kube-api-access-qsgkk\") pod \"simulator-w4-46b13b-kserve-778dd784df-c9mnx\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.186450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.186413 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:25.498230 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:05:25.496580 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod057f7e4f_090a_42ab_92a1_cc9359882375.slice/crio-95988073dd6feab4f0d8f76033a7a3f63c1cf4849a8edac50ba4bc410480597f WatchSource:0}: Error finding container 95988073dd6feab4f0d8f76033a7a3f63c1cf4849a8edac50ba4bc410480597f: Status 404 returned error can't find the container with id 95988073dd6feab4f0d8f76033a7a3f63c1cf4849a8edac50ba4bc410480597f Oct 30 14:05:25.498230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.498167 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx"] Oct 30 14:05:25.901862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.901776 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" event={"ID":"057f7e4f-090a-42ab-92a1-cc9359882375","Type":"ContainerStarted","Data":"69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630"} Oct 30 14:05:25.901862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.901821 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" event={"ID":"057f7e4f-090a-42ab-92a1-cc9359882375","Type":"ContainerStarted","Data":"95988073dd6feab4f0d8f76033a7a3f63c1cf4849a8edac50ba4bc410480597f"} Oct 30 14:05:25.962214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:25.960243 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podStartSLOduration=1.9561831920000001 podStartE2EDuration="1.956183192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:25.955605573 +0000 UTC m=+955.756111017" watchObservedRunningTime="2026-10-30 14:05:25.956183192 +0000 UTC m=+955.756688636" Oct 30 14:05:26.117337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.117291 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:05:26.117508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.117349 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 14:05:26.187281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.187250 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:26.878881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.878839 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh"] Oct 30 14:05:26.882430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.882407 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:26.904868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.904839 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh"] Oct 30 14:05:26.916261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.916232 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"default-dockercfg-lbdrq\"" Oct 30 14:05:26.916407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.916314 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"premium-w3-00926c-kserve-self-signed-certs\"" Oct 30 14:05:26.916407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.916340 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"openshift-service-ca.crt\"" Oct 30 14:05:26.916692 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.916661 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-00926c\"/\"kube-root-ca.crt\"" Oct 30 14:05:26.979995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.979963 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-model-cache\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:26.980160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.980099 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tmp-dir\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:26.980338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.980313 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nn68\" (UniqueName: \"kubernetes.io/projected/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-kube-api-access-5nn68\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:26.980420 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.980361 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-dshm\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:26.980581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.980569 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tls-certs\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:26.980703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:26.980684 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-home\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.082214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.081568 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tls-certs\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.082214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.081629 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-home\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.082214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.081693 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-model-cache\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.082214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.081760 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tmp-dir\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.082214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.081835 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nn68\" (UniqueName: \"kubernetes.io/projected/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-kube-api-access-5nn68\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.082214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.081863 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-dshm\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.083211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.082728 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-model-cache\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.085927 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.085627 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tmp-dir\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.085927 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.085665 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-home\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.087179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.087135 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tls-certs\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.090300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.089791 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-dshm\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.112363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.112337 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nn68\" (UniqueName: \"kubernetes.io/projected/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-kube-api-access-5nn68\") pod \"premium-w3-00926c-kserve-75d5d7df67-8hhgh\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.117236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.117211 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:05:27.117339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.117260 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 14:05:27.197445 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.197419 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:27.521951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.521905 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh"] Oct 30 14:05:27.522367 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:05:27.522342 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13fb459c_d0d5_4498_a63c_0b8f4d288fdf.slice/crio-6f51aba32ce1d2fc0a2a02209d461d5389978e64c4e25ee67dde9c05455c4796 WatchSource:0}: Error finding container 6f51aba32ce1d2fc0a2a02209d461d5389978e64c4e25ee67dde9c05455c4796: Status 404 returned error can't find the container with id 6f51aba32ce1d2fc0a2a02209d461d5389978e64c4e25ee67dde9c05455c4796 Oct 30 14:05:27.916431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.916337 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" event={"ID":"13fb459c-d0d5-4498-a63c-0b8f4d288fdf","Type":"ContainerStarted","Data":"39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a"} Oct 30 14:05:27.916431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.916385 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" event={"ID":"13fb459c-d0d5-4498-a63c-0b8f4d288fdf","Type":"ContainerStarted","Data":"6f51aba32ce1d2fc0a2a02209d461d5389978e64c4e25ee67dde9c05455c4796"} Oct 30 14:05:27.967782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:27.967720 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podStartSLOduration=1.9677004839999999 podStartE2EDuration="1.967700484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:27.967395904 +0000 UTC m=+957.767901349" watchObservedRunningTime="2026-10-30 14:05:27.967700484 +0000 UTC m=+957.768205929" Oct 30 14:05:28.117523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:28.117484 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:05:28.117682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:28.117548 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 14:05:28.198094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:28.198061 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:29.117256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:29.117217 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:05:29.117653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:29.117280 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 14:05:30.116879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:30.116841 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 14:05:30.117277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:30.117242 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 14:05:31.121514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:31.121474 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:31.122066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:31.121788 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:31.122577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:31.122559 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:05:33.640734 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:05:33.640696 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:05:35.187175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:35.187136 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:35.201530 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:35.201503 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:35.977432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:35.977396 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:05:37.197949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:37.197918 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:37.210217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:37.210177 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:37.989305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:37.989269 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:05:39.003688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.003652 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp"] Oct 30 14:05:39.008997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.008975 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.043001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.042970 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a7bb0c-openshift-default-dockercfg-25sds\"" Oct 30 14:05:39.045248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.045222 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp"] Oct 30 14:05:39.116294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116252 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.116294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116301 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.116466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116395 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.116466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116422 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.116466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116449 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5g2\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-kube-api-access-zk5g2\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.116592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116571 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6636d3-8077-4e87-95a3-e1325f6f677c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.116712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116696 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.116756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116726 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.116756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.116750 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.211964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.211937 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd"] Oct 30 14:05:39.215978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.215959 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.217510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217485 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.217615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217517 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5g2\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-kube-api-access-zk5g2\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.217615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217558 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6636d3-8077-4e87-95a3-e1325f6f677c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.217734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217656 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.217734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217684 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.217734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217723 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.217896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217788 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.217896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217818 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.218000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.217906 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.218321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.218298 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.218458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.218430 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.218605 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.218334 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.218727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.218650 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6636d3-8077-4e87-95a3-e1325f6f677c-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.218843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.218821 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.221232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.221208 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.221784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.221763 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.246741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.246715 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd"] Oct 30 14:05:39.253549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.253526 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.253639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.253565 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5g2\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-kube-api-access-zk5g2\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.318909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.318852 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.318909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.318903 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.319068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.318919 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.319106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.319067 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8mp\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-kube-api-access-zx8mp\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.319106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.319097 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.319176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.319120 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.319287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.319270 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.319324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.319307 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.319407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.319395 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.321684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.321668 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:39.420125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420092 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420131 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420214 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8mp\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-kube-api-access-zx8mp\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420242 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420281 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420331 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420368 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420405 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420437 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420593 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-credential-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.420949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420922 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-data\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.421066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.420952 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-certs\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.421066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.421018 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-socket\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.421271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.421250 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istiod-ca-cert\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.422612 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.422592 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-envoy\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.423409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.423386 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-podinfo\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.448266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.448239 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-token\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.448347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.448323 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8mp\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-kube-api-access-zx8mp\") pod \"e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.461131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.461106 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n"] Oct 30 14:05:39.466799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.466781 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.486429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.486407 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n"] Oct 30 14:05:39.487523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.487504 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-46b13b-openshift-default-dockercfg-922c6\"" Oct 30 14:05:39.521344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521315 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.521458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521367 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff15b799-4e23-43b3-961d-c160bf18e272-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.521526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521455 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crghx\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-kube-api-access-crghx\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.521526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521503 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.521601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521575 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.521634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521610 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.521686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521673 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.521721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521707 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.521754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.521731 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff15b799-4e23-43b3-961d-c160bf18e272-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.552110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.552088 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:39.622904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.622876 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.622949 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623011 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623047 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff15b799-4e23-43b3-961d-c160bf18e272-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623174 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623316 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff15b799-4e23-43b3-961d-c160bf18e272-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623371 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crghx\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-kube-api-access-crghx\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623406 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623437 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623452 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623742 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623798 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.623923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.623874 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff15b799-4e23-43b3-961d-c160bf18e272-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.624215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.624163 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.625823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.625803 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.626032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.626013 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff15b799-4e23-43b3-961d-c160bf18e272-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.637069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.637042 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp"] Oct 30 14:05:39.638696 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:05:39.638671 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6636d3_8077_4e87_95a3_e1325f6f677c.slice/crio-c05f3143718a84d695d7c5b40c7bd7576d33490e28d7a57dab06ad6f7e469f45 WatchSource:0}: Error finding container c05f3143718a84d695d7c5b40c7bd7576d33490e28d7a57dab06ad6f7e469f45: Status 404 returned error can't find the container with id c05f3143718a84d695d7c5b40c7bd7576d33490e28d7a57dab06ad6f7e469f45 Oct 30 14:05:39.645892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.645858 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:39.645971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.645926 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:39.645971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.645956 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:39.654531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.654506 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.654787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.654764 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crghx\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-kube-api-access-crghx\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.781012 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.780982 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:39.858993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.858955 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd"] Oct 30 14:05:39.859479 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:05:39.859446 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47280eb_1f64_4558_99bf_21d0ed2a41d1.slice/crio-02ac105c73db5e37e896714fc6435ec7fbba1c9caee4ef0d03f75cad2361b24e WatchSource:0}: Error finding container 02ac105c73db5e37e896714fc6435ec7fbba1c9caee4ef0d03f75cad2361b24e: Status 404 returned error can't find the container with id 02ac105c73db5e37e896714fc6435ec7fbba1c9caee4ef0d03f75cad2361b24e Oct 30 14:05:39.862430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.862395 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:39.862542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.862463 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:39.862542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.862503 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:39.993056 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.993021 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" event={"ID":"3c6636d3-8077-4e87-95a3-e1325f6f677c","Type":"ContainerStarted","Data":"8924a227f802e4348e80a2d26f06a1f5f3a22745b4ea46708c5b7e2e666b3159"} Oct 30 14:05:39.993281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.993082 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" event={"ID":"3c6636d3-8077-4e87-95a3-e1325f6f677c","Type":"ContainerStarted","Data":"c05f3143718a84d695d7c5b40c7bd7576d33490e28d7a57dab06ad6f7e469f45"} Oct 30 14:05:39.996240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.996215 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" event={"ID":"d47280eb-1f64-4558-99bf-21d0ed2a41d1","Type":"ContainerStarted","Data":"db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab"} Oct 30 14:05:39.996427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:39.996409 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" event={"ID":"d47280eb-1f64-4558-99bf-21d0ed2a41d1","Type":"ContainerStarted","Data":"02ac105c73db5e37e896714fc6435ec7fbba1c9caee4ef0d03f75cad2361b24e"} Oct 30 14:05:40.044595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.044535 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podStartSLOduration=2.044514624 podStartE2EDuration="2.044514624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:40.041419253 +0000 UTC m=+969.841924699" watchObservedRunningTime="2026-10-30 14:05:40.044514624 +0000 UTC m=+969.845020071" Oct 30 14:05:40.092605 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.092537 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podStartSLOduration=1.0925143290000001 podStartE2EDuration="1.092514329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:40.089796741 +0000 UTC m=+969.890302185" watchObservedRunningTime="2026-10-30 14:05:40.092514329 +0000 UTC m=+969.893019776" Oct 30 14:05:40.110908 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:05:40.110864 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff15b799_4e23_43b3_961d_c160bf18e272.slice/crio-a2d268c7d51391f7e29708d2042fd5aac1479dbcf89078dcc1b2d8f7be88a205 WatchSource:0}: Error finding container a2d268c7d51391f7e29708d2042fd5aac1479dbcf89078dcc1b2d8f7be88a205: Status 404 returned error can't find the container with id a2d268c7d51391f7e29708d2042fd5aac1479dbcf89078dcc1b2d8f7be88a205 Oct 30 14:05:40.113690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.113646 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:40.114409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.114366 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:40.114578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.114557 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:05:40.116866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.116836 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n"] Oct 30 14:05:40.322647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.322616 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:40.323749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.323723 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:05:40.323855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.323765 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 14:05:40.556214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.556136 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:40.556419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.556289 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:05:40.556419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:40.556334 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 14:05:41.024633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.024442 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" event={"ID":"ff15b799-4e23-43b3-961d-c160bf18e272","Type":"ContainerStarted","Data":"be70137b9f78d477f95d02adf4a2b7d04b00e94a390d0452e675e73701902003"} Oct 30 14:05:41.024633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.024541 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" event={"ID":"ff15b799-4e23-43b3-961d-c160bf18e272","Type":"ContainerStarted","Data":"a2d268c7d51391f7e29708d2042fd5aac1479dbcf89078dcc1b2d8f7be88a205"} Oct 30 14:05:41.088211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.081140 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podStartSLOduration=2.081121457 podStartE2EDuration="2.081121457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:41.079733111 +0000 UTC m=+970.880238596" watchObservedRunningTime="2026-10-30 14:05:41.081121457 +0000 UTC m=+970.881626905" Oct 30 14:05:41.322704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.322617 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:05:41.322704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.322678 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 14:05:41.567221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.566615 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:05:41.567221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.566680 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 14:05:41.786214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.785267 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:41.799213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.797284 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:05:41.799213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:41.797347 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 14:05:42.322723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:42.322696 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:05:42.323179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:42.323156 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 14:05:42.561213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:42.560514 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:05:42.561213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:42.560575 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 14:05:42.786608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:42.786566 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:05:42.787322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:42.786890 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 14:05:43.323011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:43.322958 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:05:43.323431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:43.323036 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 14:05:43.554799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:43.554763 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:05:43.555083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:43.555055 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 14:05:43.781585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:43.781546 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:05:43.781881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:43.781853 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 14:05:44.323211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:44.322534 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:05:44.323211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:44.322592 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 14:05:44.556718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:44.556680 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:05:44.557028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:44.556977 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 14:05:44.782938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:44.782903 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:05:44.783250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:44.783218 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 14:05:45.322551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:45.322512 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:05:45.322762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:45.322736 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 14:05:45.565215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:45.558552 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 14:05:45.565215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:45.558613 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 14:05:45.790277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:45.790240 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 14:05:45.790579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:45.790547 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 14:05:46.455591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.455542 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:46.456822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.456787 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:46.457790 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.457765 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:05:46.672131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.672104 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:46.673411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.673392 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:46.675131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.675113 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:05:46.944565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.944533 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:46.947578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.947063 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:46.947872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:46.947848 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:05:54.666567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.666531 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc"] Oct 30 14:05:54.674441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.674413 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.687758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.687730 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc"] Oct 30 14:05:54.708255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.698879 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"kube-root-ca.crt\"" Oct 30 14:05:54.708255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.699215 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"openshift-service-ca.crt\"" Oct 30 14:05:54.708255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.707661 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"default-dockercfg-hskdp\"" Oct 30 14:05:54.708255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.707694 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-da78c4\"/\"premium-w2-da78c4-kserve-self-signed-certs\"" Oct 30 14:05:54.732655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.732623 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62fn\" (UniqueName: \"kubernetes.io/projected/81b1292b-2202-4023-90a9-3409d4cf2481-kube-api-access-n62fn\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.732812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.732671 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b1292b-2202-4023-90a9-3409d4cf2481-tls-certs\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.732812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.732774 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-home\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.732888 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.732821 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-model-cache\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.732923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.732889 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-dshm\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.732923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.732916 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-tmp-dir\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.834076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.834037 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-model-cache\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.834283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.834141 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-dshm\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.834283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.834224 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-tmp-dir\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.834283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.834275 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62fn\" (UniqueName: \"kubernetes.io/projected/81b1292b-2202-4023-90a9-3409d4cf2481-kube-api-access-n62fn\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.834465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.834324 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b1292b-2202-4023-90a9-3409d4cf2481-tls-certs\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.834465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.834418 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-home\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.834575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.834535 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-model-cache\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.834810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.834767 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-home\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.835632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.835328 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-tmp-dir\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.837134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.837078 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-dshm\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.837881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.837819 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b1292b-2202-4023-90a9-3409d4cf2481-tls-certs\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.872629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.872583 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62fn\" (UniqueName: \"kubernetes.io/projected/81b1292b-2202-4023-90a9-3409d4cf2481-kube-api-access-n62fn\") pod \"premium-w2-da78c4-kserve-646b6c8fd9-csvsc\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:54.989036 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:54.988999 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:05:55.293558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:55.293530 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc"] Oct 30 14:05:55.293666 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:05:55.293528 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b1292b_2202_4023_90a9_3409d4cf2481.slice/crio-edd8147a49363e6e8b2cbcf8372b64fe5eefb60a293949c77e32baeca5806d7f WatchSource:0}: Error finding container edd8147a49363e6e8b2cbcf8372b64fe5eefb60a293949c77e32baeca5806d7f: Status 404 returned error can't find the container with id edd8147a49363e6e8b2cbcf8372b64fe5eefb60a293949c77e32baeca5806d7f Oct 30 14:05:56.145688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:56.145654 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" event={"ID":"81b1292b-2202-4023-90a9-3409d4cf2481","Type":"ContainerStarted","Data":"515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9"} Oct 30 14:05:56.145688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:56.145692 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" event={"ID":"81b1292b-2202-4023-90a9-3409d4cf2481","Type":"ContainerStarted","Data":"edd8147a49363e6e8b2cbcf8372b64fe5eefb60a293949c77e32baeca5806d7f"} Oct 30 14:05:56.191722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:56.191671 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podStartSLOduration=2.1916582 podStartE2EDuration="2.1916582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:56.190031039 +0000 UTC m=+985.990536483" watchObservedRunningTime="2026-10-30 14:05:56.1916582 +0000 UTC m=+985.992163648" Oct 30 14:05:56.989880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:05:56.989825 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:06:02.649216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.639232 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl"] Oct 30 14:06:02.721426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.721386 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl"] Oct 30 14:06:02.721779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.721757 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.791777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.791715 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"default-dockercfg-jhpvm\"" Oct 30 14:06:02.792329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.792292 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"openshift-service-ca.crt\"" Oct 30 14:06:02.792504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.792318 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"kube-root-ca.crt\"" Oct 30 14:06:02.792732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.792510 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"premium-w5-a7bb0c-kserve-self-signed-certs\"" Oct 30 14:06:02.836382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.836351 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7fh\" (UniqueName: \"kubernetes.io/projected/2230a84a-9c4e-413d-8612-87844196be0e-kube-api-access-sm7fh\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.836545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.836388 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-model-cache\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.836545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.836408 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2230a84a-9c4e-413d-8612-87844196be0e-tls-certs\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.836649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.836623 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-dshm\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.836701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.836654 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-home\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.836701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.836690 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-tmp-dir\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.937573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.937531 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-home\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.937747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.937604 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-tmp-dir\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.937747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.937678 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7fh\" (UniqueName: \"kubernetes.io/projected/2230a84a-9c4e-413d-8612-87844196be0e-kube-api-access-sm7fh\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.937747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.937707 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-model-cache\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.937747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.937731 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2230a84a-9c4e-413d-8612-87844196be0e-tls-certs\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.938019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.937942 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-dshm\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.938974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.938950 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-home\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.939339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.939314 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-model-cache\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.939722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.939703 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-tmp-dir\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.941236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.940792 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-dshm\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.942941 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.942924 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2230a84a-9c4e-413d-8612-87844196be0e-tls-certs\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:02.978996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:02.978957 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7fh\" (UniqueName: \"kubernetes.io/projected/2230a84a-9c4e-413d-8612-87844196be0e-kube-api-access-sm7fh\") pod \"premium-w5-a7bb0c-kserve-889887c7d-6s7tl\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:03.112465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:03.112418 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:03.979221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:03.975527 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl"] Oct 30 14:06:04.216995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:04.216959 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" event={"ID":"2230a84a-9c4e-413d-8612-87844196be0e","Type":"ContainerStarted","Data":"94cd90bd8e9f56d256fcb550a27784b7352ec84de25ef0c7c4739d1245876857"} Oct 30 14:06:04.217564 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:06:04.217527 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:06:04.989595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:04.989558 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:06:05.003254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:05.003220 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:06:05.227498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:05.227455 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" event={"ID":"2230a84a-9c4e-413d-8612-87844196be0e","Type":"ContainerStarted","Data":"d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006"} Oct 30 14:06:05.244804 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:05.244730 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:06:05.275445 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:05.275388 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podStartSLOduration=3.275374293 podStartE2EDuration="3.275374293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:05.272549424 +0000 UTC m=+995.073054868" watchObservedRunningTime="2026-10-30 14:06:05.275374293 +0000 UTC m=+995.075879752" Oct 30 14:06:06.114073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:06.114036 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:08.263839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.263805 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr"] Oct 30 14:06:08.269023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.269001 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.289144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.289118 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr"] Oct 30 14:06:08.291476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.291448 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eddaacf-b1dc-41be-96fb-4ca539929156-tls-certs\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.291579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.291504 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-tmp-dir\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.291642 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.291617 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-dshm\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.291686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.291664 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-home\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.291747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.291713 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9sv\" (UniqueName: \"kubernetes.io/projected/6eddaacf-b1dc-41be-96fb-4ca539929156-kube-api-access-8q9sv\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.291921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.291894 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-model-cache\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.299148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.299131 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-46b13b\"/\"premium-w4-46b13b-kserve-self-signed-certs\"" Oct 30 14:06:08.392666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.392627 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eddaacf-b1dc-41be-96fb-4ca539929156-tls-certs\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.392666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.392667 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-tmp-dir\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.392909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.392709 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-dshm\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.392909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.392736 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-home\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.392909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.392785 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9sv\" (UniqueName: \"kubernetes.io/projected/6eddaacf-b1dc-41be-96fb-4ca539929156-kube-api-access-8q9sv\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.393108 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.393085 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-tmp-dir\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.393163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.393140 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-home\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.393336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.393313 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-model-cache\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.393696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.393672 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-model-cache\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.395081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.395059 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-dshm\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.395215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.395181 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eddaacf-b1dc-41be-96fb-4ca539929156-tls-certs\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.418550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.418526 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9sv\" (UniqueName: \"kubernetes.io/projected/6eddaacf-b1dc-41be-96fb-4ca539929156-kube-api-access-8q9sv\") pod \"premium-w4-46b13b-kserve-6c64c684f6-z87tr\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.581877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.581796 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:08.883797 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:06:08.883764 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6eddaacf_b1dc_41be_96fb_4ca539929156.slice/crio-46a83b5e1455fad6255b5ee3088802cb49d8a7b30242befd0fe92feea55b5fcf WatchSource:0}: Error finding container 46a83b5e1455fad6255b5ee3088802cb49d8a7b30242befd0fe92feea55b5fcf: Status 404 returned error can't find the container with id 46a83b5e1455fad6255b5ee3088802cb49d8a7b30242befd0fe92feea55b5fcf Oct 30 14:06:08.883911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:08.883864 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr"] Oct 30 14:06:09.276657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:09.276622 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" event={"ID":"6eddaacf-b1dc-41be-96fb-4ca539929156","Type":"ContainerStarted","Data":"0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b"} Oct 30 14:06:09.277050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:09.276671 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" event={"ID":"6eddaacf-b1dc-41be-96fb-4ca539929156","Type":"ContainerStarted","Data":"46a83b5e1455fad6255b5ee3088802cb49d8a7b30242befd0fe92feea55b5fcf"} Oct 30 14:06:09.324752 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:09.324693 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podStartSLOduration=1.324674925 podStartE2EDuration="1.324674925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:09.32088995 +0000 UTC m=+999.121395395" watchObservedRunningTime="2026-10-30 14:06:09.324674925 +0000 UTC m=+999.125180373" Oct 30 14:06:09.582377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:09.582291 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:13.113697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:13.113620 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:13.126288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:13.126263 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:13.318476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:13.318449 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:06:18.582573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:18.582538 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:18.595270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:18.595248 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:19.352908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:19.352873 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:06:33.642353 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:06:33.642312 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:06:54.872883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.872840 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk"] Oct 30 14:06:54.878619 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.878587 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.898826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.898800 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk"] Oct 30 14:06:54.908934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.908909 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ca229c-openshift-default-dockercfg-52bwp\"" Oct 30 14:06:54.946731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.946702 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzkf7\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-kube-api-access-zzkf7\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.946872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.946829 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.946931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.946881 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.946974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.946963 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.947062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.947049 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d6e232-fd91-466c-befa-f8db13723638-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.947099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.947074 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.947099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.947091 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.947181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.947159 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6e232-fd91-466c-befa-f8db13723638-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:54.947293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:54.947275 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048400 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d6e232-fd91-466c-befa-f8db13723638-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048433 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048466 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048582 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6e232-fd91-466c-befa-f8db13723638-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048642 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048690 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzkf7\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-kube-api-access-zzkf7\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048774 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048822 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048848 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.048979 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.048941 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.049143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.049124 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.049249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.049220 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.049334 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.049318 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.049378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.049358 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6e232-fd91-466c-befa-f8db13723638-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.050880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.050854 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.051013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.050962 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d6e232-fd91-466c-befa-f8db13723638-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.074184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.074160 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzkf7\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-kube-api-access-zzkf7\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.074305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.074223 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.200813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.200788 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:55.517930 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:06:55.517897 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d6e232_fd91_466c_befa_f8db13723638.slice/crio-41c196d57331f3afc31c1efd39bb0bd7b07f7d685d558e1156a6f522f944239e WatchSource:0}: Error finding container 41c196d57331f3afc31c1efd39bb0bd7b07f7d685d558e1156a6f522f944239e: Status 404 returned error can't find the container with id 41c196d57331f3afc31c1efd39bb0bd7b07f7d685d558e1156a6f522f944239e Oct 30 14:06:55.518359 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.518337 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk"] Oct 30 14:06:55.520267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.520230 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:06:55.520374 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.520299 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:06:55.520374 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.520332 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:06:55.606128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.606078 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" event={"ID":"b7d6e232-fd91-466c-befa-f8db13723638","Type":"ContainerStarted","Data":"a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d"} Oct 30 14:06:55.606280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.606160 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" event={"ID":"b7d6e232-fd91-466c-befa-f8db13723638","Type":"ContainerStarted","Data":"41c196d57331f3afc31c1efd39bb0bd7b07f7d685d558e1156a6f522f944239e"} Oct 30 14:06:55.657362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:55.657306 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podStartSLOduration=1.6572889370000001 podStartE2EDuration="1.657288937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:55.654558406 +0000 UTC m=+1045.455063851" watchObservedRunningTime="2026-10-30 14:06:55.657288937 +0000 UTC m=+1045.457794382" Oct 30 14:06:56.201857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:56.201819 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:06:56.203173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:56.203139 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:06:56.203312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:56.203223 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 14:06:57.201787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:57.201755 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:06:57.201995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:57.201822 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 14:06:58.201964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:58.201930 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:06:58.202258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:58.201995 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 14:06:59.201401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:59.201369 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:06:59.201573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:06:59.201424 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 14:07:00.201925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:00.201884 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:07:00.202369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:00.201943 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 14:07:01.201561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:01.201523 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:07:01.201751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:01.201586 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 14:07:02.201230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.201156 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 14:07:02.201773 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.201266 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 14:07:02.701983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.701926 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp"] Oct 30 14:07:02.744293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.740690 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp"] Oct 30 14:07:02.744293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.740801 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:02.774675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.772603 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:07:02.774675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.773380 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:07:02.929064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.929024 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:02.929260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.929243 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:02.929364 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:02.929342 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqpt\" (UniqueName: \"kubernetes.io/projected/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-kube-api-access-hrqpt\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:03.031130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.031032 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:03.031308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.031228 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqpt\" (UniqueName: \"kubernetes.io/projected/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-kube-api-access-hrqpt\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:03.031308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.031290 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:03.031843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.031822 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:03.033638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.033620 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:03.068474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.068447 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqpt\" (UniqueName: \"kubernetes.io/projected/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-kube-api-access-hrqpt\") pod \"maas-api-revoke-sub-0f19bc01-vzpqp\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:03.143470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.143441 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:03.631015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.630622 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp"] Oct 30 14:07:03.632234 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:03.632156 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a4ac4f_fc4f_4bb5_86ad_2a644b3ce56a.slice/crio-5f5476b6d12c5bd32476b9af3e8fda85d2e4e7c946fc3c9cb64e522277275548 WatchSource:0}: Error finding container 5f5476b6d12c5bd32476b9af3e8fda85d2e4e7c946fc3c9cb64e522277275548: Status 404 returned error can't find the container with id 5f5476b6d12c5bd32476b9af3e8fda85d2e4e7c946fc3c9cb64e522277275548 Oct 30 14:07:03.699327 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:07:03.689892 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:07:03.713771 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.711290 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:07:03.713771 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.712870 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:07:03.792704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.792275 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:07:03.797767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.797738 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" event={"ID":"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a","Type":"ContainerStarted","Data":"5f5476b6d12c5bd32476b9af3e8fda85d2e4e7c946fc3c9cb64e522277275548"} Oct 30 14:07:03.917598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.917517 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6"] Oct 30 14:07:03.923859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.923831 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:03.931743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.931707 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqvw\" (UniqueName: \"kubernetes.io/projected/3c182b11-31fe-410a-b3f5-9130ddac9552-kube-api-access-dvqvw\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:03.931866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.931775 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:03.931963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.931945 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:03.936762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:03.936734 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6"] Oct 30 14:07:04.032642 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.032609 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:04.032798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.032689 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqvw\" (UniqueName: \"kubernetes.io/projected/3c182b11-31fe-410a-b3f5-9130ddac9552-kube-api-access-dvqvw\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:04.032798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.032726 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:04.033448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.033412 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:04.035026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.034992 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:04.071413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.071329 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqvw\" (UniqueName: \"kubernetes.io/projected/3c182b11-31fe-410a-b3f5-9130ddac9552-kube-api-access-dvqvw\") pod \"maas-api-revoke-sub-6d18926a-4tsw6\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:04.242519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.242489 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:04.566581 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:04.566478 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c182b11_31fe_410a_b3f5_9130ddac9552.slice/crio-5d92728a1a48c25e45f5e6efa56ce3a676108a937d9d3a5bbfda175845ad1dba WatchSource:0}: Error finding container 5d92728a1a48c25e45f5e6efa56ce3a676108a937d9d3a5bbfda175845ad1dba: Status 404 returned error can't find the container with id 5d92728a1a48c25e45f5e6efa56ce3a676108a937d9d3a5bbfda175845ad1dba Oct 30 14:07:04.568759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.568700 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6"] Oct 30 14:07:04.805907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:04.805862 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" event={"ID":"3c182b11-31fe-410a-b3f5-9130ddac9552","Type":"ContainerStarted","Data":"5d92728a1a48c25e45f5e6efa56ce3a676108a937d9d3a5bbfda175845ad1dba"} Oct 30 14:07:06.825166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:06.825074 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" event={"ID":"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a","Type":"ContainerStarted","Data":"1d6d2c48f15c081b14e768fbd7a8f02fe9a11ee642521408e287301886020501"} Oct 30 14:07:06.826719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:06.826694 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" event={"ID":"3c182b11-31fe-410a-b3f5-9130ddac9552","Type":"ContainerStarted","Data":"52afe3b07dc1b7893b65668631b54f425c4e7bcb95cd87d2c14254823268c5c4"} Oct 30 14:07:06.869807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:06.869745 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" podStartSLOduration=2.562787468 podStartE2EDuration="4.869728132s" totalImagesPullingTime="2.306940664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:06.868732853 +0000 UTC m=+1056.669238300" watchObservedRunningTime="2026-10-30 14:07:06.869728132 +0000 UTC m=+1056.670233579" Oct 30 14:07:06.913789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:06.913736 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" podStartSLOduration=2.4525327089999998 podStartE2EDuration="3.913720772s" totalImagesPullingTime="1.461188063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:06.912230007 +0000 UTC m=+1056.712735445" watchObservedRunningTime="2026-10-30 14:07:06.913720772 +0000 UTC m=+1056.714226216" Oct 30 14:07:07.834299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:07.834259 5824 generic.go:356] "Generic (PLEG): container finished" podID="f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a" containerID="1d6d2c48f15c081b14e768fbd7a8f02fe9a11ee642521408e287301886020501" exitCode=0 Oct 30 14:07:07.834757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:07.834346 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" event={"ID":"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a","Type":"ContainerDied","Data":"1d6d2c48f15c081b14e768fbd7a8f02fe9a11ee642521408e287301886020501"} Oct 30 14:07:07.836126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:07.836093 5824 generic.go:356] "Generic (PLEG): container finished" podID="3c182b11-31fe-410a-b3f5-9130ddac9552" containerID="52afe3b07dc1b7893b65668631b54f425c4e7bcb95cd87d2c14254823268c5c4" exitCode=0 Oct 30 14:07:07.836248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:07.836146 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" event={"ID":"3c182b11-31fe-410a-b3f5-9130ddac9552","Type":"ContainerDied","Data":"52afe3b07dc1b7893b65668631b54f425c4e7bcb95cd87d2c14254823268c5c4"} Oct 30 14:07:09.054399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.054371 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:09.057750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.057717 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:09.112608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.112526 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-kube-api-access-hrqpt\" (UniqueName: \"kubernetes.io/projected/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-kube-api-access-hrqpt\") pod \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " Oct 30 14:07:09.112608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.112594 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c182b11-31fe-410a-b3f5-9130ddac9552-kube-api-access-dvqvw\" (UniqueName: \"kubernetes.io/projected/3c182b11-31fe-410a-b3f5-9130ddac9552-kube-api-access-dvqvw\") pod \"3c182b11-31fe-410a-b3f5-9130ddac9552\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " Oct 30 14:07:09.112852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.112654 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-xks-ca\") pod \"3c182b11-31fe-410a-b3f5-9130ddac9552\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " Oct 30 14:07:09.112852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.112702 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-xks-ca\") pod \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " Oct 30 14:07:09.112852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.112821 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-openshift-ca\") pod \"3c182b11-31fe-410a-b3f5-9130ddac9552\" (UID: \"3c182b11-31fe-410a-b3f5-9130ddac9552\") " Oct 30 14:07:09.113025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.112917 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-openshift-ca\") pod \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\" (UID: \"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a\") " Oct 30 14:07:09.113396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.113362 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-openshift-ca" pod "3c182b11-31fe-410a-b3f5-9130ddac9552" (UID: "3c182b11-31fe-410a-b3f5-9130ddac9552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:09.113794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.113758 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-openshift-ca" pod "f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a" (UID: "f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:09.117544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.117521 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-xks-ca" pod "3c182b11-31fe-410a-b3f5-9130ddac9552" (UID: "3c182b11-31fe-410a-b3f5-9130ddac9552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:09.117711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.117692 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c182b11-31fe-410a-b3f5-9130ddac9552-kube-api-access-dvqvw" pod "3c182b11-31fe-410a-b3f5-9130ddac9552" (UID: "3c182b11-31fe-410a-b3f5-9130ddac9552"). InnerVolumeSpecName "kube-api-access-dvqvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:09.117791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.117705 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-xks-ca" pod "f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a" (UID: "f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:09.117992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.117972 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-kube-api-access-hrqpt" pod "f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a" (UID: "f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a"). InnerVolumeSpecName "kube-api-access-hrqpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:09.214294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.214258 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrqpt\" (UniqueName: \"kubernetes.io/projected/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-kube-api-access-hrqpt\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.214294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.214287 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqvw\" (UniqueName: \"kubernetes.io/projected/3c182b11-31fe-410a-b3f5-9130ddac9552-kube-api-access-dvqvw\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.214294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.214298 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.214550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.214309 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.214550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.214317 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c182b11-31fe-410a-b3f5-9130ddac9552-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.214550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.214327 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:09.849256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.849221 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" Oct 30 14:07:09.849445 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.849224 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6" event={"ID":"3c182b11-31fe-410a-b3f5-9130ddac9552","Type":"ContainerDied","Data":"5d92728a1a48c25e45f5e6efa56ce3a676108a937d9d3a5bbfda175845ad1dba"} Oct 30 14:07:09.849445 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.849351 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d92728a1a48c25e45f5e6efa56ce3a676108a937d9d3a5bbfda175845ad1dba" Oct 30 14:07:09.850598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.850576 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" Oct 30 14:07:09.850598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.850591 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp" event={"ID":"f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a","Type":"ContainerDied","Data":"5f5476b6d12c5bd32476b9af3e8fda85d2e4e7c946fc3c9cb64e522277275548"} Oct 30 14:07:09.850779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:09.850631 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f5476b6d12c5bd32476b9af3e8fda85d2e4e7c946fc3c9cb64e522277275548" Oct 30 14:07:13.127390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.127351 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn"] Oct 30 14:07:13.127984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.127968 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3c182b11-31fe-410a-b3f5-9130ddac9552" containerName="revoke-keys" Oct 30 14:07:13.128028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.127986 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c182b11-31fe-410a-b3f5-9130ddac9552" containerName="revoke-keys" Oct 30 14:07:13.128028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.128019 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a" containerName="revoke-keys" Oct 30 14:07:13.128028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.128025 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a" containerName="revoke-keys" Oct 30 14:07:13.128129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.128119 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a" containerName="revoke-keys" Oct 30 14:07:13.128163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.128131 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c182b11-31fe-410a-b3f5-9130ddac9552" containerName="revoke-keys" Oct 30 14:07:13.131890 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.131873 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.149765 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.149730 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn"] Oct 30 14:07:13.160170 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.160137 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:07:13.160331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.160183 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:07:13.267155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.267124 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.267365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.267333 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xn5\" (UniqueName: \"kubernetes.io/projected/0e376886-9e11-4ed3-a546-1249ec38b620-kube-api-access-z8xn5\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.267426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.267380 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.368531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.368491 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xn5\" (UniqueName: \"kubernetes.io/projected/0e376886-9e11-4ed3-a546-1249ec38b620-kube-api-access-z8xn5\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.368731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.368541 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.368731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.368616 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.369322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.369302 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.371036 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.371005 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.395486 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.395424 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xn5\" (UniqueName: \"kubernetes.io/projected/0e376886-9e11-4ed3-a546-1249ec38b620-kube-api-access-z8xn5\") pod \"maas-api-revoke-sub-cbe6e47c-l66dn\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.446046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.446013 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:13.756906 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:13.756873 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e376886_9e11_4ed3_a546_1249ec38b620.slice/crio-943e2ec5eb441171218f5ccca1f6fe0615dacdf164e4493de3d11b43497d0c80 WatchSource:0}: Error finding container 943e2ec5eb441171218f5ccca1f6fe0615dacdf164e4493de3d11b43497d0c80: Status 404 returned error can't find the container with id 943e2ec5eb441171218f5ccca1f6fe0615dacdf164e4493de3d11b43497d0c80 Oct 30 14:07:13.757884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.757850 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn"] Oct 30 14:07:13.876405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:13.876362 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" event={"ID":"0e376886-9e11-4ed3-a546-1249ec38b620","Type":"ContainerStarted","Data":"943e2ec5eb441171218f5ccca1f6fe0615dacdf164e4493de3d11b43497d0c80"} Oct 30 14:07:14.883584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:14.883542 5824 generic.go:356] "Generic (PLEG): container finished" podID="0e376886-9e11-4ed3-a546-1249ec38b620" containerID="fb4608ef495921b27e2b02b8730c4893ef64b5551cc6b50a73b771997e872a36" exitCode=0 Oct 30 14:07:14.883967 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:14.883634 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" event={"ID":"0e376886-9e11-4ed3-a546-1249ec38b620","Type":"ContainerDied","Data":"fb4608ef495921b27e2b02b8730c4893ef64b5551cc6b50a73b771997e872a36"} Oct 30 14:07:16.441281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.440883 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:16.511733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.507621 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-openshift-ca\") pod \"0e376886-9e11-4ed3-a546-1249ec38b620\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " Oct 30 14:07:16.511733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.507720 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-xks-ca\") pod \"0e376886-9e11-4ed3-a546-1249ec38b620\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " Oct 30 14:07:16.511733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.507796 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e376886-9e11-4ed3-a546-1249ec38b620-kube-api-access-z8xn5\" (UniqueName: \"kubernetes.io/projected/0e376886-9e11-4ed3-a546-1249ec38b620-kube-api-access-z8xn5\") pod \"0e376886-9e11-4ed3-a546-1249ec38b620\" (UID: \"0e376886-9e11-4ed3-a546-1249ec38b620\") " Oct 30 14:07:16.511733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.511681 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-openshift-ca" pod "0e376886-9e11-4ed3-a546-1249ec38b620" (UID: "0e376886-9e11-4ed3-a546-1249ec38b620"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:16.519426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.519384 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-xks-ca" pod "0e376886-9e11-4ed3-a546-1249ec38b620" (UID: "0e376886-9e11-4ed3-a546-1249ec38b620"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:16.569478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.538500 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e376886-9e11-4ed3-a546-1249ec38b620-kube-api-access-z8xn5" pod "0e376886-9e11-4ed3-a546-1249ec38b620" (UID: "0e376886-9e11-4ed3-a546-1249ec38b620"). InnerVolumeSpecName "kube-api-access-z8xn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:16.656213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.655636 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:16.656213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.655679 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e376886-9e11-4ed3-a546-1249ec38b620-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:16.656213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.655694 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xn5\" (UniqueName: \"kubernetes.io/projected/0e376886-9e11-4ed3-a546-1249ec38b620-kube-api-access-z8xn5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:16.913876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.913783 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" event={"ID":"0e376886-9e11-4ed3-a546-1249ec38b620","Type":"ContainerDied","Data":"943e2ec5eb441171218f5ccca1f6fe0615dacdf164e4493de3d11b43497d0c80"} Oct 30 14:07:16.914121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.914101 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="943e2ec5eb441171218f5ccca1f6fe0615dacdf164e4493de3d11b43497d0c80" Oct 30 14:07:16.918319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:16.918296 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn" Oct 30 14:07:17.757018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:17.756321 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:07:32.782128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.782090 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96"] Oct 30 14:07:32.782984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.782958 5824 cpu_manager.go:414] "containerMap: removing container" podUID="0e376886-9e11-4ed3-a546-1249ec38b620" containerName="revoke-keys" Oct 30 14:07:32.782984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.782981 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e376886-9e11-4ed3-a546-1249ec38b620" containerName="revoke-keys" Oct 30 14:07:32.783212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.783102 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e376886-9e11-4ed3-a546-1249ec38b620" containerName="revoke-keys" Oct 30 14:07:32.786842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.786822 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.807304 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.807272 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96"] Oct 30 14:07:32.817152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.817109 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-46b13b\"/\"distinct-w4-46b13b-kserve-self-signed-certs\"" Oct 30 14:07:32.887867 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.887812 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-dshm\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.888035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.887941 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-model-cache\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.888035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.887975 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-home\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.888035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.888030 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349413fe-1608-4cd2-90d0-3dca7efc3606-tls-certs\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.888139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.888098 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-tmp-dir\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.888139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.888127 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjsk8\" (UniqueName: \"kubernetes.io/projected/349413fe-1608-4cd2-90d0-3dca7efc3606-kube-api-access-kjsk8\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.989699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.989664 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-dshm\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.989917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.989720 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-model-cache\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.989917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.989835 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-home\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.989917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.989906 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349413fe-1608-4cd2-90d0-3dca7efc3606-tls-certs\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.990094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.989992 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-tmp-dir\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.990094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.990035 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsk8\" (UniqueName: \"kubernetes.io/projected/349413fe-1608-4cd2-90d0-3dca7efc3606-kube-api-access-kjsk8\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.990233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.990176 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-model-cache\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.990922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.990890 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-tmp-dir\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.991216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.991173 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-home\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.992548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.992520 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-dshm\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:32.993492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:32.993460 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349413fe-1608-4cd2-90d0-3dca7efc3606-tls-certs\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:33.016532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.016504 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjsk8\" (UniqueName: \"kubernetes.io/projected/349413fe-1608-4cd2-90d0-3dca7efc3606-kube-api-access-kjsk8\") pod \"distinct-w4-46b13b-kserve-7f9d98b55f-xfk96\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:33.103384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.103288 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:33.110807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.110774 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx"] Oct 30 14:07:33.118448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.115442 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.145246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.142264 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx"] Oct 30 14:07:33.193620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.193154 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.193620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.193228 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf4m4\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-kube-api-access-bf4m4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.193620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.193353 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.193620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.193382 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.193620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.193413 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.193620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.193518 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.194581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.194231 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c60bc55-ac39-410b-be5f-247da3bca3fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.194581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.194307 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.194581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.194344 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295462 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c60bc55-ac39-410b-be5f-247da3bca3fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295512 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295535 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295597 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295627 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf4m4\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-kube-api-access-bf4m4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295728 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295754 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295785 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.295915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295893 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.296112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.295967 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.296112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.296022 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.296112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.296067 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.296308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.296241 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.296308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.296283 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c60bc55-ac39-410b-be5f-247da3bca3fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.298352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.298326 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.298595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.298576 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.323862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.323830 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.324013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.323833 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf4m4\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-kube-api-access-bf4m4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t5tsx\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.438512 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:33.438475 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod349413fe_1608_4cd2_90d0_3dca7efc3606.slice/crio-08eefea885521322752d40e347f760a254811186cca6fd400ca24670d378ca34 WatchSource:0}: Error finding container 08eefea885521322752d40e347f760a254811186cca6fd400ca24670d378ca34: Status 404 returned error can't find the container with id 08eefea885521322752d40e347f760a254811186cca6fd400ca24670d378ca34 Oct 30 14:07:33.439158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.439136 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96"] Oct 30 14:07:33.470047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.470023 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:33.640530 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:07:33.640494 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:07:33.793733 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:33.793702 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c60bc55_ac39_410b_be5f_247da3bca3fb.slice/crio-39d1025a7cd12a58e04d6fc20de7a4fdd9330b76ed0737bb92865d97faa59704 WatchSource:0}: Error finding container 39d1025a7cd12a58e04d6fc20de7a4fdd9330b76ed0737bb92865d97faa59704: Status 404 returned error can't find the container with id 39d1025a7cd12a58e04d6fc20de7a4fdd9330b76ed0737bb92865d97faa59704 Oct 30 14:07:33.794572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.794550 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx"] Oct 30 14:07:33.796371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.796339 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:33.796438 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.796407 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:33.796474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:33.796434 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:34.052142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.052050 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" event={"ID":"349413fe-1608-4cd2-90d0-3dca7efc3606","Type":"ContainerStarted","Data":"2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec"} Oct 30 14:07:34.052142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.052104 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" event={"ID":"349413fe-1608-4cd2-90d0-3dca7efc3606","Type":"ContainerStarted","Data":"08eefea885521322752d40e347f760a254811186cca6fd400ca24670d378ca34"} Oct 30 14:07:34.053688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.053662 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" event={"ID":"7c60bc55-ac39-410b-be5f-247da3bca3fb","Type":"ContainerStarted","Data":"ba27233119cd9df1ea668ba28cd50f080614765b1607ff91336c6f053721de19"} Oct 30 14:07:34.053831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.053740 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" event={"ID":"7c60bc55-ac39-410b-be5f-247da3bca3fb","Type":"ContainerStarted","Data":"39d1025a7cd12a58e04d6fc20de7a4fdd9330b76ed0737bb92865d97faa59704"} Oct 30 14:07:34.100632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.100578 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podStartSLOduration=2.1005653029999998 podStartE2EDuration="2.100565303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:34.099021542 +0000 UTC m=+1083.899526976" watchObservedRunningTime="2026-10-30 14:07:34.100565303 +0000 UTC m=+1083.901070746" Oct 30 14:07:34.103590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.103562 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:34.153546 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.153498 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podStartSLOduration=1.153485079 podStartE2EDuration="1.153485079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:34.151945523 +0000 UTC m=+1083.952450967" watchObservedRunningTime="2026-10-30 14:07:34.153485079 +0000 UTC m=+1083.953990580" Oct 30 14:07:34.472733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.472696 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:34.474952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.474911 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:07:34.475241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:34.475208 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 14:07:35.473238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:35.472618 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:07:35.473238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:35.472684 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 14:07:36.470943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:36.470907 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:07:36.471180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:36.470975 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 14:07:37.472699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:37.472659 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:07:37.473365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:37.473321 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 14:07:38.471342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:38.471312 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:07:38.471598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:38.471559 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 14:07:39.267338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.265615 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl"] Oct 30 14:07:39.274273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.274247 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.286140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.286116 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl"] Oct 30 14:07:39.401867 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.401832 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.402031 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.401912 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4w9\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-kube-api-access-jf4w9\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.402073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.402055 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.402124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.402112 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be00760-ece6-49c2-97f9-26a0daab728e-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.402585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.402490 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.402714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.402648 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.402779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.402744 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.402833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.402780 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.402957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.402936 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be00760-ece6-49c2-97f9-26a0daab728e-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.473254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.471579 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:07:39.473254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.471652 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 14:07:39.504566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.504535 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4w9\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-kube-api-access-jf4w9\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.504685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.504591 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.504685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.504668 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be00760-ece6-49c2-97f9-26a0daab728e-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.504808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.504791 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.504890 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.504864 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.504942 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.504990 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.505041 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be00760-ece6-49c2-97f9-26a0daab728e-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.505347 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.505414 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.505444 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.505517 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.505635 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be00760-ece6-49c2-97f9-26a0daab728e-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.505802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.505786 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.507427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.507402 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.507502 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.507451 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be00760-ece6-49c2-97f9-26a0daab728e-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.530272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.530223 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4w9\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-kube-api-access-jf4w9\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.530635 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.530613 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.591315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.591258 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:39.682526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.681245 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd"] Oct 30 14:07:39.696224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.695469 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.711351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.711300 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd"] Oct 30 14:07:39.810056 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.809967 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.810056 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.810041 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.810317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.810069 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.810317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.810096 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.810317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.810120 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.810317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.810145 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jkv\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-kube-api-access-29jkv\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.810317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.810273 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.810570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.810340 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.810570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.810396 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.911703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.911673 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.911855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.911721 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.911900 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.911844 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.911974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.911960 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912041 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912068 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912078 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912089 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912134 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912164 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912204 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jkv\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-kube-api-access-29jkv\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912408 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912494 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.912590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.912572 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.914290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.914271 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.914457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.914439 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.938175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.938150 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jkv\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-kube-api-access-29jkv\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.938290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.938151 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:39.943689 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:39.943658 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be00760_ece6_49c2_97f9_26a0daab728e.slice/crio-595b5d77397728315150577411fb1eb0c5b7d885a0dc18dd1fbb3cc8275818c6 WatchSource:0}: Error finding container 595b5d77397728315150577411fb1eb0c5b7d885a0dc18dd1fbb3cc8275818c6: Status 404 returned error can't find the container with id 595b5d77397728315150577411fb1eb0c5b7d885a0dc18dd1fbb3cc8275818c6 Oct 30 14:07:39.943912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.943894 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl"] Oct 30 14:07:39.946313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.946281 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:39.946387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.946346 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:39.946387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:39.946372 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:40.014149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.014122 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:40.110733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.110575 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" event={"ID":"6be00760-ece6-49c2-97f9-26a0daab728e","Type":"ContainerStarted","Data":"b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e"} Oct 30 14:07:40.110733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.110646 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" event={"ID":"6be00760-ece6-49c2-97f9-26a0daab728e","Type":"ContainerStarted","Data":"595b5d77397728315150577411fb1eb0c5b7d885a0dc18dd1fbb3cc8275818c6"} Oct 30 14:07:40.188212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.174746 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podStartSLOduration=1.1747279019999999 podStartE2EDuration="1.174727902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:40.173388157 +0000 UTC m=+1089.973893616" watchObservedRunningTime="2026-10-30 14:07:40.174727902 +0000 UTC m=+1089.975233352" Oct 30 14:07:40.359969 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:40.359932 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a4cb10_fc4e_4fa3_9ad9_977a6e5a499d.slice/crio-31440b566443f7a0af599d7e28b84f71cbcdc81f1f37cb16bfc5fd2566e3213d WatchSource:0}: Error finding container 31440b566443f7a0af599d7e28b84f71cbcdc81f1f37cb16bfc5fd2566e3213d: Status 404 returned error can't find the container with id 31440b566443f7a0af599d7e28b84f71cbcdc81f1f37cb16bfc5fd2566e3213d Oct 30 14:07:40.361102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.361080 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd"] Oct 30 14:07:40.363250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.363181 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:40.363357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.363293 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:40.363357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.363340 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:40.471383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.471351 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:07:40.471547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.471406 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 14:07:40.592039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:40.592005 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:41.120305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:41.120219 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" event={"ID":"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d","Type":"ContainerStarted","Data":"89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9"} Oct 30 14:07:41.120476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:41.120310 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" event={"ID":"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d","Type":"ContainerStarted","Data":"31440b566443f7a0af599d7e28b84f71cbcdc81f1f37cb16bfc5fd2566e3213d"} Oct 30 14:07:41.191217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:41.189029 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podStartSLOduration=2.189008625 podStartE2EDuration="2.189008625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:41.186751452 +0000 UTC m=+1090.987256891" watchObservedRunningTime="2026-10-30 14:07:41.189008625 +0000 UTC m=+1090.989514075" Oct 30 14:07:41.471216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:41.470730 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 14:07:41.471216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:41.470790 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 14:07:41.604213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:41.600397 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:07:41.604213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:41.600479 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 14:07:42.015209 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:42.014613 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:42.016636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:42.016610 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:07:42.016801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:42.016777 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 14:07:42.601217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:42.598676 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:07:42.601217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:42.598756 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 14:07:43.014584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.014541 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:07:43.014804 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.014617 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 14:07:43.108217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.107291 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:43.122405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.122380 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:43.208533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.208503 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:07:43.218293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.218269 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:43.218716 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.218631 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:43.219768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.219749 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:07:43.596226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.595562 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:07:43.596226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:43.595642 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 14:07:44.053218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:44.052735 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:07:44.053218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:44.052796 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 14:07:44.593622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:44.593586 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:07:44.593910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:44.593878 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 14:07:45.014616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:45.014569 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:07:45.014829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:45.014639 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 14:07:45.599226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:45.595632 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:07:45.599226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:45.595718 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 14:07:46.023146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:46.022935 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:07:46.023777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:46.023571 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 14:07:46.592539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:46.592502 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 14:07:46.592751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:46.592568 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 14:07:47.014651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:47.014608 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 14:07:47.015049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:47.014689 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 14:07:47.596092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:47.596062 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:47.596406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:47.596381 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:47.597140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:47.597119 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:07:48.019908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:48.019880 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:48.020395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:48.020303 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:48.020853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:48.020835 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:07:50.649713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.649676 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j"] Oct 30 14:07:50.654000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.653981 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.673474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.673443 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j"] Oct 30 14:07:50.687403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.686147 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:07:50.687403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.686162 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:07:50.741595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.741568 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.741732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.741637 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.741788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.741749 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9b7\" (UniqueName: \"kubernetes.io/projected/01af172e-407c-4f7c-9bdd-e892982339ec-kube-api-access-2r9b7\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.843456 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.843424 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.843636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.843504 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9b7\" (UniqueName: \"kubernetes.io/projected/01af172e-407c-4f7c-9bdd-e892982339ec-kube-api-access-2r9b7\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.843636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.843630 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.844327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.844296 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.845753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.845737 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.874460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.874431 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9b7\" (UniqueName: \"kubernetes.io/projected/01af172e-407c-4f7c-9bdd-e892982339ec-kube-api-access-2r9b7\") pod \"maas-api-revoke-sub-6c0aec2f-mcc7j\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:50.968665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:50.968641 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:51.292014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:51.291988 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j"] Oct 30 14:07:51.292014 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:51.291997 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01af172e_407c_4f7c_9bdd_e892982339ec.slice/crio-3f4274a43f26c079a22fd80bd82ba1f9b2a281f0b5c5aee4c231c6f9a0951ff3 WatchSource:0}: Error finding container 3f4274a43f26c079a22fd80bd82ba1f9b2a281f0b5c5aee4c231c6f9a0951ff3: Status 404 returned error can't find the container with id 3f4274a43f26c079a22fd80bd82ba1f9b2a281f0b5c5aee4c231c6f9a0951ff3 Oct 30 14:07:52.086779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.086749 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p"] Oct 30 14:07:52.091965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.091938 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.108672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.108644 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p"] Oct 30 14:07:52.120354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.120318 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"distinct-w5-a7bb0c-kserve-self-signed-certs\"" Oct 30 14:07:52.260055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.260021 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-dshm\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.260578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.260545 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tls-certs\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.260734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.260609 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrgf\" (UniqueName: \"kubernetes.io/projected/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-kube-api-access-mrrgf\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.260815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.260731 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-home\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.260815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.260762 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-model-cache\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.260922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.260831 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tmp-dir\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.264542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.264515 5824 generic.go:356] "Generic (PLEG): container finished" podID="01af172e-407c-4f7c-9bdd-e892982339ec" containerID="90c72a297487c73051e0a598d7c1b1895360fe4aedae7c1c8baf6850e36d6226" exitCode=0 Oct 30 14:07:52.264682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.264570 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" event={"ID":"01af172e-407c-4f7c-9bdd-e892982339ec","Type":"ContainerDied","Data":"90c72a297487c73051e0a598d7c1b1895360fe4aedae7c1c8baf6850e36d6226"} Oct 30 14:07:52.264682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.264611 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" event={"ID":"01af172e-407c-4f7c-9bdd-e892982339ec","Type":"ContainerStarted","Data":"3f4274a43f26c079a22fd80bd82ba1f9b2a281f0b5c5aee4c231c6f9a0951ff3"} Oct 30 14:07:52.365934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.365825 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-dshm\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.366156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.365980 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tls-certs\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.366156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.366039 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrgf\" (UniqueName: \"kubernetes.io/projected/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-kube-api-access-mrrgf\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.366397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.366209 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-home\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.366505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.366488 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-model-cache\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.366874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.366845 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-home\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.367005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.366985 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tmp-dir\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.367076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.367026 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tmp-dir\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.367378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.367356 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-model-cache\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.369645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.369618 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tls-certs\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.369743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.369706 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-dshm\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.393535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.393495 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrgf\" (UniqueName: \"kubernetes.io/projected/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-kube-api-access-mrrgf\") pod \"distinct-w5-a7bb0c-kserve-5648d69f49-p279p\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.467490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.467455 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:52.772446 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:52.772416 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc45f55e6_f5fe_4103_88a5_3684bfd25ccf.slice/crio-901f31ca7b0389834b3b9f3e6b666230bda5b54c8cc1618afe72c68b52742a43 WatchSource:0}: Error finding container 901f31ca7b0389834b3b9f3e6b666230bda5b54c8cc1618afe72c68b52742a43: Status 404 returned error can't find the container with id 901f31ca7b0389834b3b9f3e6b666230bda5b54c8cc1618afe72c68b52742a43 Oct 30 14:07:52.774837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:52.774813 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p"] Oct 30 14:07:53.272967 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.272924 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" event={"ID":"c45f55e6-f5fe-4103-88a5-3684bfd25ccf","Type":"ContainerStarted","Data":"c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c"} Oct 30 14:07:53.273508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.272981 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" event={"ID":"c45f55e6-f5fe-4103-88a5-3684bfd25ccf","Type":"ContainerStarted","Data":"901f31ca7b0389834b3b9f3e6b666230bda5b54c8cc1618afe72c68b52742a43"} Oct 30 14:07:53.321704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.321649 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podStartSLOduration=1.321634153 podStartE2EDuration="1.321634153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:53.318831973 +0000 UTC m=+1103.119337418" watchObservedRunningTime="2026-10-30 14:07:53.321634153 +0000 UTC m=+1103.122139598" Oct 30 14:07:53.443437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.443395 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:53.468270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.468241 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:07:53.585243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.585147 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-openshift-ca\") pod \"01af172e-407c-4f7c-9bdd-e892982339ec\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " Oct 30 14:07:53.585405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.585275 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01af172e-407c-4f7c-9bdd-e892982339ec-kube-api-access-2r9b7\" (UniqueName: \"kubernetes.io/projected/01af172e-407c-4f7c-9bdd-e892982339ec-kube-api-access-2r9b7\") pod \"01af172e-407c-4f7c-9bdd-e892982339ec\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " Oct 30 14:07:53.585405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.585325 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-xks-ca\") pod \"01af172e-407c-4f7c-9bdd-e892982339ec\" (UID: \"01af172e-407c-4f7c-9bdd-e892982339ec\") " Oct 30 14:07:53.585568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.585538 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-openshift-ca" pod "01af172e-407c-4f7c-9bdd-e892982339ec" (UID: "01af172e-407c-4f7c-9bdd-e892982339ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:07:53.585901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.585880 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:53.587392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.587369 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-xks-ca" pod "01af172e-407c-4f7c-9bdd-e892982339ec" (UID: "01af172e-407c-4f7c-9bdd-e892982339ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:07:53.587485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.587459 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01af172e-407c-4f7c-9bdd-e892982339ec-kube-api-access-2r9b7" pod "01af172e-407c-4f7c-9bdd-e892982339ec" (UID: "01af172e-407c-4f7c-9bdd-e892982339ec"). InnerVolumeSpecName "kube-api-access-2r9b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:07:53.687028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.686995 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01af172e-407c-4f7c-9bdd-e892982339ec-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:53.687146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:53.687032 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r9b7\" (UniqueName: \"kubernetes.io/projected/01af172e-407c-4f7c-9bdd-e892982339ec-kube-api-access-2r9b7\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:07:54.280234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.280202 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" Oct 30 14:07:54.280633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.280228 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j" event={"ID":"01af172e-407c-4f7c-9bdd-e892982339ec","Type":"ContainerDied","Data":"3f4274a43f26c079a22fd80bd82ba1f9b2a281f0b5c5aee4c231c6f9a0951ff3"} Oct 30 14:07:54.280633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.280278 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f4274a43f26c079a22fd80bd82ba1f9b2a281f0b5c5aee4c231c6f9a0951ff3" Oct 30 14:07:54.900247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.900208 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25"] Oct 30 14:07:54.900895 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.900874 5824 cpu_manager.go:414] "containerMap: removing container" podUID="01af172e-407c-4f7c-9bdd-e892982339ec" containerName="revoke-keys" Oct 30 14:07:54.900997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.900895 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01af172e-407c-4f7c-9bdd-e892982339ec" containerName="revoke-keys" Oct 30 14:07:54.901078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.901064 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="01af172e-407c-4f7c-9bdd-e892982339ec" containerName="revoke-keys" Oct 30 14:07:54.904830 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.904809 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:54.928829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:54.928798 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25"] Oct 30 14:07:55.003745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.003706 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.003745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.003740 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629544b6-29fe-475f-8833-bca9cca08552-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.004093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.003767 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.004093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.003885 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.004093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.003930 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629544b6-29fe-475f-8833-bca9cca08552-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.004093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.004073 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqtn\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-kube-api-access-gjqtn\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.004353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.004178 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.004353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.004297 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.004440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.004401 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.105789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.105754 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.105975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.105820 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.105975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.105845 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629544b6-29fe-475f-8833-bca9cca08552-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.105975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.105881 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.105975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.105930 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106069 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629544b6-29fe-475f-8833-bca9cca08552-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106177 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqtn\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-kube-api-access-gjqtn\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106247 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106271 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106353 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106380 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106530 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106490 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106608 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.106788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.106768 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629544b6-29fe-475f-8833-bca9cca08552-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.108471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.108447 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.108595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.108561 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629544b6-29fe-475f-8833-bca9cca08552-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.132964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.132934 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.133076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.133060 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqtn\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-kube-api-access-gjqtn\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.220280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.220255 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:55.542257 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:07:55.542218 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629544b6_29fe_475f_8833_bca9cca08552.slice/crio-76f14cfe045de65565f1803cca893a93e584fe6d2379d37d190c6a2ace42ba16 WatchSource:0}: Error finding container 76f14cfe045de65565f1803cca893a93e584fe6d2379d37d190c6a2ace42ba16: Status 404 returned error can't find the container with id 76f14cfe045de65565f1803cca893a93e584fe6d2379d37d190c6a2ace42ba16 Oct 30 14:07:55.543559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.543531 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25"] Oct 30 14:07:55.545236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.545184 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:55.545326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.545274 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:55.545326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:55.545302 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:07:56.301238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:56.301173 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" event={"ID":"629544b6-29fe-475f-8833-bca9cca08552","Type":"ContainerStarted","Data":"05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573"} Oct 30 14:07:56.301441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:56.301255 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" event={"ID":"629544b6-29fe-475f-8833-bca9cca08552","Type":"ContainerStarted","Data":"76f14cfe045de65565f1803cca893a93e584fe6d2379d37d190c6a2ace42ba16"} Oct 30 14:07:56.390723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:56.390663 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podStartSLOduration=2.3906447760000002 podStartE2EDuration="2.390644776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:56.374813785 +0000 UTC m=+1106.175319229" watchObservedRunningTime="2026-10-30 14:07:56.390644776 +0000 UTC m=+1106.191150219" Oct 30 14:07:57.221048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:57.221019 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:07:57.222703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:57.222678 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 14:07:57.222832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:57.222732 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 14:07:58.220595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:58.220561 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 14:07:58.220787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:58.220616 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 14:07:59.221535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:59.221497 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 14:07:59.222014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:07:59.221573 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 14:08:00.221496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:00.221462 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 14:08:00.221689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:00.221517 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 14:08:01.106097 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.106052 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt"] Oct 30 14:08:01.110644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.110615 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.130682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.130651 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt"] Oct 30 14:08:01.140103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.140075 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-46b13b\"/\"distinct-2-w4-46b13b-kserve-self-signed-certs\"" Oct 30 14:08:01.210946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.210912 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-home\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.210946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.210942 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-model-cache\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.211129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.210973 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-dshm\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.211256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.211239 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tls-certs\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.211297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.211269 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tmp-dir\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.211330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.211303 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wk2\" (UniqueName: \"kubernetes.io/projected/4459dc7d-8724-4e45-a3fd-d95cc04365fd-kube-api-access-f2wk2\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.220608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.220580 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 14:08:01.220720 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.220635 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 14:08:01.312415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.312381 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tmp-dir\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.312874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.312440 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wk2\" (UniqueName: \"kubernetes.io/projected/4459dc7d-8724-4e45-a3fd-d95cc04365fd-kube-api-access-f2wk2\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.312874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.312495 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-home\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.312874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.312512 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-model-cache\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.312874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.312540 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-dshm\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.312874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.312725 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tls-certs\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.313139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.312926 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tmp-dir\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.313139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.313000 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-model-cache\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.313237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.313141 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-home\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.315132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.315109 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-dshm\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.315757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.315738 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tls-certs\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.341232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.341144 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wk2\" (UniqueName: \"kubernetes.io/projected/4459dc7d-8724-4e45-a3fd-d95cc04365fd-kube-api-access-f2wk2\") pod \"distinct-2-w4-46b13b-kserve-898c49c88-kq6wt\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.428907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.428882 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:01.772026 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:08:01.771993 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4459dc7d_8724_4e45_a3fd_d95cc04365fd.slice/crio-3eabe30698e3da8d851539a629b1d1f21109d638d9d68cfca3548a401d4ba303 WatchSource:0}: Error finding container 3eabe30698e3da8d851539a629b1d1f21109d638d9d68cfca3548a401d4ba303: Status 404 returned error can't find the container with id 3eabe30698e3da8d851539a629b1d1f21109d638d9d68cfca3548a401d4ba303 Oct 30 14:08:01.774208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:01.774134 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt"] Oct 30 14:08:02.221140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:02.221101 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 14:08:02.221342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:02.221163 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 14:08:02.361529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:02.361492 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" event={"ID":"4459dc7d-8724-4e45-a3fd-d95cc04365fd","Type":"ContainerStarted","Data":"1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777"} Oct 30 14:08:02.361529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:02.361538 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" event={"ID":"4459dc7d-8724-4e45-a3fd-d95cc04365fd","Type":"ContainerStarted","Data":"3eabe30698e3da8d851539a629b1d1f21109d638d9d68cfca3548a401d4ba303"} Oct 30 14:08:02.408992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:02.408932 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podStartSLOduration=1.408913609 podStartE2EDuration="1.408913609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:02.406290179 +0000 UTC m=+1112.206795622" watchObservedRunningTime="2026-10-30 14:08:02.408913609 +0000 UTC m=+1112.209419055" Oct 30 14:08:02.429161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:02.429122 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:02.468446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:02.468419 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:08:02.481227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:02.481156 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:08:03.225610 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:03.225577 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:08:03.225874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:03.225851 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:08:03.226679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:03.226657 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:08:03.390704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:03.390675 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:08:03.641135 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:08:03.641049 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:08:11.429125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:11.429088 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:11.441282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:11.441262 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:11.526250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:11.526185 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:08:20.500799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.500759 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz"] Oct 30 14:08:20.504964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.504939 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.527768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.527738 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz"] Oct 30 14:08:20.535982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.535963 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a7bb0c\"/\"distinct-2-w5-a7bb0c-kserve-self-signed-certs\"" Oct 30 14:08:20.562114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.562090 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-model-cache\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.562244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.562134 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-dshm\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.562320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.562248 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tls-certs\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.562383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.562332 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-home\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.562445 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.562392 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tmp-dir\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.562594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.562576 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cpc\" (UniqueName: \"kubernetes.io/projected/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-kube-api-access-s5cpc\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.663960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.663925 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-model-cache\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.664125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.664007 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-dshm\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.664125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.664040 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tls-certs\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.664278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.664125 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-home\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.664278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.664147 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tmp-dir\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.664386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.664280 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cpc\" (UniqueName: \"kubernetes.io/projected/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-kube-api-access-s5cpc\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.664444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.664413 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-model-cache\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.664713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.664689 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-home\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.664799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.664712 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tmp-dir\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.666857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.666831 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-dshm\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.667051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.667034 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tls-certs\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.693381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.693353 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cpc\" (UniqueName: \"kubernetes.io/projected/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-kube-api-access-s5cpc\") pod \"distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:20.820032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:20.819943 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:21.127542 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:08:21.127504 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20a4a5cf_d7f8_4243_a491_71c65e5ed8ff.slice/crio-a5bb95494337078fe4866c0e3920c2dc74ecb599443c6478275b9d37a8aa8c24 WatchSource:0}: Error finding container a5bb95494337078fe4866c0e3920c2dc74ecb599443c6478275b9d37a8aa8c24: Status 404 returned error can't find the container with id a5bb95494337078fe4866c0e3920c2dc74ecb599443c6478275b9d37a8aa8c24 Oct 30 14:08:21.129741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:21.129716 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz"] Oct 30 14:08:21.583588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:21.583559 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" event={"ID":"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff","Type":"ContainerStarted","Data":"af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775"} Oct 30 14:08:21.583988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:21.583604 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" event={"ID":"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff","Type":"ContainerStarted","Data":"a5bb95494337078fe4866c0e3920c2dc74ecb599443c6478275b9d37a8aa8c24"} Oct 30 14:08:21.632489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:21.632441 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podStartSLOduration=1.632425792 podStartE2EDuration="1.632425792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:21.630268842 +0000 UTC m=+1131.430774299" watchObservedRunningTime="2026-10-30 14:08:21.632425792 +0000 UTC m=+1131.432931238" Oct 30 14:08:21.820025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:21.819996 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:24.514566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.514532 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj"] Oct 30 14:08:24.519613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.519538 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.554299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.554268 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj"] Oct 30 14:08:24.558360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.557615 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74"] Oct 30 14:08:24.562517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.562499 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.581532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.581508 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74"] Oct 30 14:08:24.611473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.611449 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.611616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.611558 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.611696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.611680 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqkp\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-kube-api-access-rzqkp\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.611738 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.611707 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.611738 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.611725 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.611842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.611743 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.611842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.611818 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.611980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.611962 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.612065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.612050 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.713599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.713565 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.713788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.713630 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5l9m\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-kube-api-access-r5l9m\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.713788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.713674 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.713788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.713706 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.713905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.713779 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.713905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.713831 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.713905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.713869 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.714051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.713930 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.714131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714097 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.714293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714176 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714176 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714300 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.714425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714376 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714415 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714406 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714420 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.714579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714542 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqkp\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-kube-api-access-rzqkp\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714580 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714623 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714661 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.714853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714791 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714888 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714872 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.714997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.714975 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.716535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.716515 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.716833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.716812 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.741203 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.741156 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.741342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.741205 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqkp\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-kube-api-access-rzqkp\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.816030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.815961 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5l9m\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-kube-api-access-r5l9m\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816022 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816076 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816101 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816151 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816237 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816288 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816381 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816445 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816498 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-data\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816651 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-certs\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816748 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-credential-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816809 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-socket\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.816870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.816846 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istiod-ca-cert\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.818297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.818277 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-envoy\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.818972 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.818949 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-podinfo\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.834936 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.834912 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:24.843044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.843012 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-token\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.843160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.843143 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5l9m\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-kube-api-access-r5l9m\") pod \"e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:24.881607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:24.881440 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:25.155311 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:08:25.155282 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c14891_26f7_43e9_8c8a_ff9e5fc3c735.slice/crio-576b94774c13abf1b98e35e944db29f2904fcd762eff6ea1938688dbf327f116 WatchSource:0}: Error finding container 576b94774c13abf1b98e35e944db29f2904fcd762eff6ea1938688dbf327f116: Status 404 returned error can't find the container with id 576b94774c13abf1b98e35e944db29f2904fcd762eff6ea1938688dbf327f116 Oct 30 14:08:25.156754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.156733 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj"] Oct 30 14:08:25.157855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.157823 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:25.157911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.157887 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:25.157943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.157916 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:25.199968 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:08:25.199939 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e049a0_9ed9_49c5_83bf_54f76f39ee44.slice/crio-0f473dbf0b1c5b557b55cf737a68205edd421eee7f5ef9398ed93dcb549e368e WatchSource:0}: Error finding container 0f473dbf0b1c5b557b55cf737a68205edd421eee7f5ef9398ed93dcb549e368e: Status 404 returned error can't find the container with id 0f473dbf0b1c5b557b55cf737a68205edd421eee7f5ef9398ed93dcb549e368e Oct 30 14:08:25.200288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.200263 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74"] Oct 30 14:08:25.202765 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.202727 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:25.202854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.202794 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:25.202854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.202844 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:25.616327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.616280 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" event={"ID":"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735","Type":"ContainerStarted","Data":"afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833"} Oct 30 14:08:25.616826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.616378 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" event={"ID":"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735","Type":"ContainerStarted","Data":"576b94774c13abf1b98e35e944db29f2904fcd762eff6ea1938688dbf327f116"} Oct 30 14:08:25.618289 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.618257 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" event={"ID":"97e049a0-9ed9-49c5-83bf-54f76f39ee44","Type":"ContainerStarted","Data":"175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d"} Oct 30 14:08:25.618434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.618313 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" event={"ID":"97e049a0-9ed9-49c5-83bf-54f76f39ee44","Type":"ContainerStarted","Data":"0f473dbf0b1c5b557b55cf737a68205edd421eee7f5ef9398ed93dcb549e368e"} Oct 30 14:08:25.721757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.721700 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podStartSLOduration=1.721686158 podStartE2EDuration="1.721686158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:25.669977603 +0000 UTC m=+1135.470483050" watchObservedRunningTime="2026-10-30 14:08:25.721686158 +0000 UTC m=+1135.522191602" Oct 30 14:08:25.723013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.722981 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podStartSLOduration=1.722971409 podStartE2EDuration="1.722971409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:25.721042235 +0000 UTC m=+1135.521547679" watchObservedRunningTime="2026-10-30 14:08:25.722971409 +0000 UTC m=+1135.523476853" Oct 30 14:08:25.835988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.835954 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:25.889366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:25.885296 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:26.837622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:26.837585 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:08:26.837999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:26.837654 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 14:08:26.891851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:26.891814 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 14:08:26.892152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:26.892121 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 14:08:27.835330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:27.835297 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:08:27.835511 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:27.835359 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 14:08:27.888216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:27.886604 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 14:08:27.888216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:27.886681 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 14:08:28.835915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:28.835879 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:08:28.836093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:28.835939 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 14:08:28.883214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:28.882802 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 14:08:28.883214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:28.882863 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 14:08:29.835508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:29.835465 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:08:29.835987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:29.835534 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 14:08:29.884214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:29.883684 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 14:08:29.884214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:29.883757 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 14:08:30.820298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:30.820267 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:30.835528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:30.835492 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:08:30.837470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:30.835544 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 14:08:30.838028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:30.838008 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:30.895148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:30.895109 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 14:08:30.895440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:30.895412 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 14:08:31.730661 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:31.730626 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:08:31.835564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:31.835527 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 14:08:31.835953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:31.835593 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 14:08:31.882784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:31.882749 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 14:08:31.882969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:31.882812 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 14:08:32.839965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:32.839934 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:32.840515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:32.840492 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:32.841141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:32.841116 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:08:32.886504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:32.886476 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:32.889054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:32.888351 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:32.897932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:32.897902 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:08:33.658819 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:08:33.658770 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:08:54.045043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.045003 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d"] Oct 30 14:08:54.051307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.051277 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.069868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.067549 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d"] Oct 30 14:08:54.075843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.075812 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:08:54.076001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.075980 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:08:54.099854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.095817 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.099854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.096248 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.099854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.096430 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzzs\" (UniqueName: \"kubernetes.io/projected/90ec0077-ba94-4422-83eb-fc127f893b07-kube-api-access-gfzzs\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.197653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.197613 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.197851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.197691 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.197851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.197735 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzzs\" (UniqueName: \"kubernetes.io/projected/90ec0077-ba94-4422-83eb-fc127f893b07-kube-api-access-gfzzs\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.198330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.198307 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.200138 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.200116 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.227076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.227041 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzzs\" (UniqueName: \"kubernetes.io/projected/90ec0077-ba94-4422-83eb-fc127f893b07-kube-api-access-gfzzs\") pod \"maas-api-revoke-sub-0590961f-n7n6d\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.373098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.373016 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:54.712089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.710999 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d"] Oct 30 14:08:54.715808 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:08:54.715696 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ec0077_ba94_4422_83eb_fc127f893b07.slice/crio-6a2853befa3d31ccc04885c427745e542970b5dc0b47f02fb919f3c3065d80e9 WatchSource:0}: Error finding container 6a2853befa3d31ccc04885c427745e542970b5dc0b47f02fb919f3c3065d80e9: Status 404 returned error can't find the container with id 6a2853befa3d31ccc04885c427745e542970b5dc0b47f02fb919f3c3065d80e9 Oct 30 14:08:54.904947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.904895 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" event={"ID":"90ec0077-ba94-4422-83eb-fc127f893b07","Type":"ContainerStarted","Data":"6a2853befa3d31ccc04885c427745e542970b5dc0b47f02fb919f3c3065d80e9"} Oct 30 14:08:54.929736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.929696 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv"] Oct 30 14:08:54.937254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.937223 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:54.958214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:54.956296 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv"] Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.026167 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.026733 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.026801 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cm67\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-kube-api-access-5cm67\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.026837 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.026908 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.026942 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.027031 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c61044-6bb0-41a0-9655-588f8dff0470-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.027119 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.027270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.027168 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.129472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.129426 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c61044-6bb0-41a0-9655-588f8dff0470-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.129937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.129552 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.129937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.129602 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.129937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.129665 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.129937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.129763 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.129937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.129826 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cm67\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-kube-api-access-5cm67\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.129937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.129861 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.130346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.130243 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-certs\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.130409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.130346 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.130409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.130393 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.130629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.130609 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.130718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.130649 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c61044-6bb0-41a0-9655-588f8dff0470-istiod-ca-cert\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.130953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.130929 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-credential-socket\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.131055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.131028 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-data\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.142478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.142423 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-envoy\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.142768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.142743 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-podinfo\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.158139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.158108 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-token\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.158385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.158361 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cm67\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-kube-api-access-5cm67\") pod \"e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.260258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.260233 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:55.693224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.692057 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv"] Oct 30 14:08:55.723708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.722442 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:55.723708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.722541 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:55.723708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.722595 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:08:55.915470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.915386 5824 generic.go:356] "Generic (PLEG): container finished" podID="90ec0077-ba94-4422-83eb-fc127f893b07" containerID="020db3c1c355b492c31c86dc00f851d8aea64003eb3c44dadd4f9c216fe9853f" exitCode=0 Oct 30 14:08:55.915628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.915468 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" event={"ID":"90ec0077-ba94-4422-83eb-fc127f893b07","Type":"ContainerDied","Data":"020db3c1c355b492c31c86dc00f851d8aea64003eb3c44dadd4f9c216fe9853f"} Oct 30 14:08:55.918265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.918239 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" event={"ID":"f4c61044-6bb0-41a0-9655-588f8dff0470","Type":"ContainerStarted","Data":"1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf"} Oct 30 14:08:55.918376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:55.918282 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" event={"ID":"f4c61044-6bb0-41a0-9655-588f8dff0470","Type":"ContainerStarted","Data":"78460645670f8a075c96d496abd32d61aa6d0e0c79e417ecbe98f7509dd8e396"} Oct 30 14:08:56.011525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:56.011473 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podStartSLOduration=2.011459333 podStartE2EDuration="2.011459333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:56.008638433 +0000 UTC m=+1165.809143876" watchObservedRunningTime="2026-10-30 14:08:56.011459333 +0000 UTC m=+1165.811964776" Oct 30 14:08:56.261008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:56.260951 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:08:56.263000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:56.262972 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 14:08:56.263228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:56.263035 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 14:08:57.115909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.115887 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:57.200303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.200180 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-xks-ca\") pod \"90ec0077-ba94-4422-83eb-fc127f893b07\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " Oct 30 14:08:57.200459 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.200316 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ec0077-ba94-4422-83eb-fc127f893b07-kube-api-access-gfzzs\" (UniqueName: \"kubernetes.io/projected/90ec0077-ba94-4422-83eb-fc127f893b07-kube-api-access-gfzzs\") pod \"90ec0077-ba94-4422-83eb-fc127f893b07\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " Oct 30 14:08:57.200459 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.200410 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-openshift-ca\") pod \"90ec0077-ba94-4422-83eb-fc127f893b07\" (UID: \"90ec0077-ba94-4422-83eb-fc127f893b07\") " Oct 30 14:08:57.201002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.200970 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-openshift-ca" pod "90ec0077-ba94-4422-83eb-fc127f893b07" (UID: "90ec0077-ba94-4422-83eb-fc127f893b07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:08:57.201123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.201103 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:08:57.203088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.203045 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ec0077-ba94-4422-83eb-fc127f893b07-kube-api-access-gfzzs" pod "90ec0077-ba94-4422-83eb-fc127f893b07" (UID: "90ec0077-ba94-4422-83eb-fc127f893b07"). InnerVolumeSpecName "kube-api-access-gfzzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:08:57.213498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.213473 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-xks-ca" pod "90ec0077-ba94-4422-83eb-fc127f893b07" (UID: "90ec0077-ba94-4422-83eb-fc127f893b07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:08:57.266593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.266542 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 14:08:57.266996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.266611 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 14:08:57.302389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.302358 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec0077-ba94-4422-83eb-fc127f893b07-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:08:57.302389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.302382 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfzzs\" (UniqueName: \"kubernetes.io/projected/90ec0077-ba94-4422-83eb-fc127f893b07-kube-api-access-gfzzs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:08:57.933133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.933101 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" Oct 30 14:08:57.933133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.933126 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d" event={"ID":"90ec0077-ba94-4422-83eb-fc127f893b07","Type":"ContainerDied","Data":"6a2853befa3d31ccc04885c427745e542970b5dc0b47f02fb919f3c3065d80e9"} Oct 30 14:08:57.933352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:57.933171 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a2853befa3d31ccc04885c427745e542970b5dc0b47f02fb919f3c3065d80e9" Oct 30 14:08:58.266616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:58.266521 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 14:08:58.266616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:58.266577 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 14:08:59.266606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:59.266564 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 14:08:59.266815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:08:59.266625 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 14:09:00.262184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:00.262150 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 14:09:00.262469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:00.262441 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 14:09:01.263214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:01.262498 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 14:09:01.263214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:01.262571 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 14:09:02.261544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:02.261498 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 14:09:02.261734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:02.261580 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 14:09:03.264857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:03.264823 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:09:03.265293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:03.265089 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:09:03.265873 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:03.265857 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:09:03.644437 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:09:03.644347 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:09:07.726317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:07.726271 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:18.394372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.394287 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j"] Oct 30 14:09:18.395154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.395129 5824 cpu_manager.go:414] "containerMap: removing container" podUID="90ec0077-ba94-4422-83eb-fc127f893b07" containerName="revoke-keys" Oct 30 14:09:18.395154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.395154 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ec0077-ba94-4422-83eb-fc127f893b07" containerName="revoke-keys" Oct 30 14:09:18.395412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.395396 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ec0077-ba94-4422-83eb-fc127f893b07" containerName="revoke-keys" Oct 30 14:09:18.399134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.399111 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.441011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.440975 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j"] Oct 30 14:09:18.446092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.446065 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:09:18.446290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.446177 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:09:18.491552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.491515 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59wrf\" (UniqueName: \"kubernetes.io/projected/9387165a-731d-44fc-a638-203ed61da74f-kube-api-access-59wrf\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.491723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.491576 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9387165a-731d-44fc-a638-203ed61da74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.491723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.491705 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9387165a-731d-44fc-a638-203ed61da74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.593837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.593802 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9387165a-731d-44fc-a638-203ed61da74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.594003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.593974 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59wrf\" (UniqueName: \"kubernetes.io/projected/9387165a-731d-44fc-a638-203ed61da74f-kube-api-access-59wrf\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.594051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.594011 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9387165a-731d-44fc-a638-203ed61da74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.594588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.594565 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9387165a-731d-44fc-a638-203ed61da74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.596275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.596248 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9387165a-731d-44fc-a638-203ed61da74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.620547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.620519 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59wrf\" (UniqueName: \"kubernetes.io/projected/9387165a-731d-44fc-a638-203ed61da74f-kube-api-access-59wrf\") pod \"maas-api-revoke-sub-4535894a-rkc2j\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:18.742540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:18.742504 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:19.048294 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:09:19.048250 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9387165a_731d_44fc_a638_203ed61da74f.slice/crio-dabfc210dda8ce545689bfc09f965c1217f7c1e77e124f71e8ed887fb89eb655 WatchSource:0}: Error finding container dabfc210dda8ce545689bfc09f965c1217f7c1e77e124f71e8ed887fb89eb655: Status 404 returned error can't find the container with id dabfc210dda8ce545689bfc09f965c1217f7c1e77e124f71e8ed887fb89eb655 Oct 30 14:09:19.049053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:19.049021 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j"] Oct 30 14:09:19.130143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:19.130111 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" event={"ID":"9387165a-731d-44fc-a638-203ed61da74f","Type":"ContainerStarted","Data":"dabfc210dda8ce545689bfc09f965c1217f7c1e77e124f71e8ed887fb89eb655"} Oct 30 14:09:20.139017 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:20.138981 5824 generic.go:356] "Generic (PLEG): container finished" podID="9387165a-731d-44fc-a638-203ed61da74f" containerID="601127f7bcdc26a32f778c53e8283fa7b814afd1e73c4de03736125c13193d84" exitCode=0 Oct 30 14:09:20.139453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:20.139026 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" event={"ID":"9387165a-731d-44fc-a638-203ed61da74f","Type":"ContainerDied","Data":"601127f7bcdc26a32f778c53e8283fa7b814afd1e73c4de03736125c13193d84"} Oct 30 14:09:21.300152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.300127 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:21.436422 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.436390 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9387165a-731d-44fc-a638-203ed61da74f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9387165a-731d-44fc-a638-203ed61da74f-maas-api-xks-ca\") pod \"9387165a-731d-44fc-a638-203ed61da74f\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " Oct 30 14:09:21.436594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.436501 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9387165a-731d-44fc-a638-203ed61da74f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9387165a-731d-44fc-a638-203ed61da74f-maas-api-openshift-ca\") pod \"9387165a-731d-44fc-a638-203ed61da74f\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " Oct 30 14:09:21.436594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.436560 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9387165a-731d-44fc-a638-203ed61da74f-kube-api-access-59wrf\" (UniqueName: \"kubernetes.io/projected/9387165a-731d-44fc-a638-203ed61da74f-kube-api-access-59wrf\") pod \"9387165a-731d-44fc-a638-203ed61da74f\" (UID: \"9387165a-731d-44fc-a638-203ed61da74f\") " Oct 30 14:09:21.436894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.436867 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9387165a-731d-44fc-a638-203ed61da74f-maas-api-openshift-ca" pod "9387165a-731d-44fc-a638-203ed61da74f" (UID: "9387165a-731d-44fc-a638-203ed61da74f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:21.437646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.437622 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9387165a-731d-44fc-a638-203ed61da74f-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:21.438569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.438547 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9387165a-731d-44fc-a638-203ed61da74f-maas-api-xks-ca" pod "9387165a-731d-44fc-a638-203ed61da74f" (UID: "9387165a-731d-44fc-a638-203ed61da74f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:09:21.438829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.438801 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9387165a-731d-44fc-a638-203ed61da74f-kube-api-access-59wrf" pod "9387165a-731d-44fc-a638-203ed61da74f" (UID: "9387165a-731d-44fc-a638-203ed61da74f"). InnerVolumeSpecName "kube-api-access-59wrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:21.539556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.539511 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9387165a-731d-44fc-a638-203ed61da74f-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:21.539556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:21.539541 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59wrf\" (UniqueName: \"kubernetes.io/projected/9387165a-731d-44fc-a638-203ed61da74f-kube-api-access-59wrf\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:22.154217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:22.154162 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" Oct 30 14:09:22.154217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:22.154197 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j" event={"ID":"9387165a-731d-44fc-a638-203ed61da74f","Type":"ContainerDied","Data":"dabfc210dda8ce545689bfc09f965c1217f7c1e77e124f71e8ed887fb89eb655"} Oct 30 14:09:22.154437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:22.154257 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dabfc210dda8ce545689bfc09f965c1217f7c1e77e124f71e8ed887fb89eb655" Oct 30 14:09:25.213780 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.213748 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr"] Oct 30 14:09:25.214579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.214561 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9387165a-731d-44fc-a638-203ed61da74f" containerName="revoke-keys" Oct 30 14:09:25.214647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.214581 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9387165a-731d-44fc-a638-203ed61da74f" containerName="revoke-keys" Oct 30 14:09:25.214722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.214712 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9387165a-731d-44fc-a638-203ed61da74f" containerName="revoke-keys" Oct 30 14:09:25.218928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.218899 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.253969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.253929 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr"] Oct 30 14:09:25.287797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.287763 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6z7\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-kube-api-access-8s6z7\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.288072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.288043 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.288235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.288135 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.288235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.288178 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.288386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.288320 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.288386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.288349 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.288386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.288371 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.288542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.288414 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.288542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.288436 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390140 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390183 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390239 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390267 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390284 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390379 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6z7\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-kube-api-access-8s6z7\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390418 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390502 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390552 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.390750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.390715 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-data\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.391075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.391046 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-certs\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.391269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.391239 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.391371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.391351 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-credential-socket\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.391429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.391374 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istiod-ca-cert\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.392769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.392745 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-envoy\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.392854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.392807 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-podinfo\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.421639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.421615 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-token\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.421785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.421759 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6z7\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-kube-api-access-8s6z7\") pod \"e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.562426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.562377 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:25.884837 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:09:25.884808 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2336d1_a75c_4870_a2ba_918cd6662c6e.slice/crio-cd18291e418426a499d4a596cbc5e1687e8a90944ba3a10e137d9bd4f6296713 WatchSource:0}: Error finding container cd18291e418426a499d4a596cbc5e1687e8a90944ba3a10e137d9bd4f6296713: Status 404 returned error can't find the container with id cd18291e418426a499d4a596cbc5e1687e8a90944ba3a10e137d9bd4f6296713 Oct 30 14:09:25.886097 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.886076 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr"] Oct 30 14:09:25.888245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.888172 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:09:25.888347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.888279 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:09:25.888347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:25.888313 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:09:26.196234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:26.196171 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" event={"ID":"3c2336d1-a75c-4870-a2ba-918cd6662c6e","Type":"ContainerStarted","Data":"a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c"} Oct 30 14:09:26.196525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:26.196494 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" event={"ID":"3c2336d1-a75c-4870-a2ba-918cd6662c6e","Type":"ContainerStarted","Data":"cd18291e418426a499d4a596cbc5e1687e8a90944ba3a10e137d9bd4f6296713"} Oct 30 14:09:26.266723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:26.266663 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podStartSLOduration=1.2666431519999999 podStartE2EDuration="1.266643152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:26.261884088 +0000 UTC m=+1196.062389533" watchObservedRunningTime="2026-10-30 14:09:26.266643152 +0000 UTC m=+1196.067148599" Oct 30 14:09:26.564270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:26.563533 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:26.565667 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:26.565510 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:09:26.565667 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:26.565570 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 14:09:27.564609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:27.564569 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:09:27.565250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:27.565205 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 14:09:28.568820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:28.568785 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:09:28.569434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:28.569400 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 14:09:29.564213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:29.563565 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:09:29.564213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:29.563643 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 14:09:30.567214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:30.566301 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:09:30.567214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:30.566378 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 14:09:30.971224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:30.971160 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d"] Oct 30 14:09:30.981228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:30.979542 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d"] Oct 30 14:09:30.981228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:30.979657 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:30.997604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:30.997579 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:09:30.997777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:30.997756 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:09:31.088658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.088617 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.088849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.088747 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.088849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.088832 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lhn\" (UniqueName: \"kubernetes.io/projected/c8f86e69-b76f-4c1e-a145-578dd368f5b5-kube-api-access-w7lhn\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.190812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.190777 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.190989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.190872 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.190989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.190954 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lhn\" (UniqueName: \"kubernetes.io/projected/c8f86e69-b76f-4c1e-a145-578dd368f5b5-kube-api-access-w7lhn\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.191520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.191494 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.193236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.193217 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.217419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.217393 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lhn\" (UniqueName: \"kubernetes.io/projected/c8f86e69-b76f-4c1e-a145-578dd368f5b5-kube-api-access-w7lhn\") pod \"maas-api-revoke-sub-efa74430-sp54d\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.297793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.297710 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:31.567314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.566037 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:09:31.567314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.566110 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 14:09:31.635461 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:09:31.635428 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f86e69_b76f_4c1e_a145_578dd368f5b5.slice/crio-85d627a8b242f1eb7165ef1e2a2034045264c30fe9ef5bc7eefa443e12673a42 WatchSource:0}: Error finding container 85d627a8b242f1eb7165ef1e2a2034045264c30fe9ef5bc7eefa443e12673a42: Status 404 returned error can't find the container with id 85d627a8b242f1eb7165ef1e2a2034045264c30fe9ef5bc7eefa443e12673a42 Oct 30 14:09:31.636324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:31.636295 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d"] Oct 30 14:09:32.306712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:32.306676 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" event={"ID":"c8f86e69-b76f-4c1e-a145-578dd368f5b5","Type":"ContainerStarted","Data":"3a1f54c266d3b3f70ef2bbc2062b8e1346db87425e314582e31e0d9d7462a42a"} Oct 30 14:09:32.306929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:32.306740 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" event={"ID":"c8f86e69-b76f-4c1e-a145-578dd368f5b5","Type":"ContainerStarted","Data":"85d627a8b242f1eb7165ef1e2a2034045264c30fe9ef5bc7eefa443e12673a42"} Oct 30 14:09:32.352801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:32.352733 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" podStartSLOduration=2.3527149339999998 podStartE2EDuration="2.352714934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:32.350127114 +0000 UTC m=+1202.150632561" watchObservedRunningTime="2026-10-30 14:09:32.352714934 +0000 UTC m=+1202.153220379" Oct 30 14:09:32.563464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:32.563384 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 14:09:32.563464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:32.563438 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 14:09:33.315352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:33.315313 5824 generic.go:356] "Generic (PLEG): container finished" podID="c8f86e69-b76f-4c1e-a145-578dd368f5b5" containerID="3a1f54c266d3b3f70ef2bbc2062b8e1346db87425e314582e31e0d9d7462a42a" exitCode=0 Oct 30 14:09:33.315934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:33.315403 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" event={"ID":"c8f86e69-b76f-4c1e-a145-578dd368f5b5","Type":"ContainerDied","Data":"3a1f54c266d3b3f70ef2bbc2062b8e1346db87425e314582e31e0d9d7462a42a"} Oct 30 14:09:33.642891 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:09:33.642805 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:09:34.527503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.527472 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:34.608363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.608269 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:09:34.608528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.608350 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:34.662929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.662892 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-xks-ca\") pod \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " Oct 30 14:09:34.663139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.663081 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f86e69-b76f-4c1e-a145-578dd368f5b5-kube-api-access-w7lhn\" (UniqueName: \"kubernetes.io/projected/c8f86e69-b76f-4c1e-a145-578dd368f5b5-kube-api-access-w7lhn\") pod \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " Oct 30 14:09:34.663139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.663125 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-openshift-ca\") pod \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\" (UID: \"c8f86e69-b76f-4c1e-a145-578dd368f5b5\") " Oct 30 14:09:34.663772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.663741 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-openshift-ca" pod "c8f86e69-b76f-4c1e-a145-578dd368f5b5" (UID: "c8f86e69-b76f-4c1e-a145-578dd368f5b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:34.663965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.663943 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:34.665640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.665603 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-xks-ca" pod "c8f86e69-b76f-4c1e-a145-578dd368f5b5" (UID: "c8f86e69-b76f-4c1e-a145-578dd368f5b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:09:34.666283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.666255 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f86e69-b76f-4c1e-a145-578dd368f5b5-kube-api-access-w7lhn" pod "c8f86e69-b76f-4c1e-a145-578dd368f5b5" (UID: "c8f86e69-b76f-4c1e-a145-578dd368f5b5"). InnerVolumeSpecName "kube-api-access-w7lhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:34.765474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.765431 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f86e69-b76f-4c1e-a145-578dd368f5b5-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:34.765474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:34.765477 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7lhn\" (UniqueName: \"kubernetes.io/projected/c8f86e69-b76f-4c1e-a145-578dd368f5b5-kube-api-access-w7lhn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:35.015093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:35.015061 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:35.015472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:35.015443 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:35.016612 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:35.016585 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:09:35.331552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:35.331482 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" Oct 30 14:09:35.331552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:35.331522 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d" event={"ID":"c8f86e69-b76f-4c1e-a145-578dd368f5b5","Type":"ContainerDied","Data":"85d627a8b242f1eb7165ef1e2a2034045264c30fe9ef5bc7eefa443e12673a42"} Oct 30 14:09:35.331724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:35.331570 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d627a8b242f1eb7165ef1e2a2034045264c30fe9ef5bc7eefa443e12673a42" Oct 30 14:09:36.889343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.889308 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp"] Oct 30 14:09:36.889988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.889974 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c8f86e69-b76f-4c1e-a145-578dd368f5b5" containerName="revoke-keys" Oct 30 14:09:36.890040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.889989 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f86e69-b76f-4c1e-a145-578dd368f5b5" containerName="revoke-keys" Oct 30 14:09:36.890087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.890077 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f86e69-b76f-4c1e-a145-578dd368f5b5" containerName="revoke-keys" Oct 30 14:09:36.893848 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.893826 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:36.910753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.910723 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp"] Oct 30 14:09:36.920319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.920292 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:09:36.920457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.920357 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:09:36.999770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.999740 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:36.999950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.999792 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:36.999950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:36.999927 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vh6b\" (UniqueName: \"kubernetes.io/projected/1f8f2355-d43f-4945-986b-0528d04e18cc-kube-api-access-4vh6b\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:37.101388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.101351 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:37.101581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.101408 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:37.101581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.101451 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vh6b\" (UniqueName: \"kubernetes.io/projected/1f8f2355-d43f-4945-986b-0528d04e18cc-kube-api-access-4vh6b\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:37.102252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.102225 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:37.103788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.103765 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:37.128179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.128145 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vh6b\" (UniqueName: \"kubernetes.io/projected/1f8f2355-d43f-4945-986b-0528d04e18cc-kube-api-access-4vh6b\") pod \"maas-api-revoke-sub-96d2aa13-bmnsp\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:37.209099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.209066 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:37.519456 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.518837 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp"] Oct 30 14:09:37.522475 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:09:37.522445 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8f2355_d43f_4945_986b_0528d04e18cc.slice/crio-66c82217f2746fcef2cf9df02e9aa311415e83b6e8f69ed1b83cfc7a827e792d WatchSource:0}: Error finding container 66c82217f2746fcef2cf9df02e9aa311415e83b6e8f69ed1b83cfc7a827e792d: Status 404 returned error can't find the container with id 66c82217f2746fcef2cf9df02e9aa311415e83b6e8f69ed1b83cfc7a827e792d Oct 30 14:09:37.563612 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.563586 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl"] Oct 30 14:09:37.575987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.575960 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl"] Oct 30 14:09:37.576132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.576066 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.594276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.594250 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"default-dockercfg-499ms\"" Oct 30 14:09:37.594500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.594313 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"distinct-w1-ca229c-kserve-self-signed-certs\"" Oct 30 14:09:37.594500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.594409 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"kube-root-ca.crt\"" Oct 30 14:09:37.607590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.607566 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca229c\"/\"openshift-service-ca.crt\"" Oct 30 14:09:37.710607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.710563 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-home\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.710803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.710692 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4kj6\" (UniqueName: \"kubernetes.io/projected/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-kube-api-access-v4kj6\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.710904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.710836 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tls-certs\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.710904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.710881 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-model-cache\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.711052 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.710941 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-dshm\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.711111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.711068 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tmp-dir\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.812421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.812326 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-home\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.812600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.812422 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4kj6\" (UniqueName: \"kubernetes.io/projected/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-kube-api-access-v4kj6\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.812600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.812569 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tls-certs\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.812737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.812620 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-model-cache\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.812737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.812691 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-dshm\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.812837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.812795 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-home\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.812893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.812837 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tmp-dir\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.813171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.813098 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tmp-dir\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.813171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.813140 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-model-cache\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.814991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.814968 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-dshm\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.815253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.815236 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tls-certs\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.841394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.841360 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4kj6\" (UniqueName: \"kubernetes.io/projected/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-kube-api-access-v4kj6\") pod \"distinct-w1-ca229c-kserve-6f95878d64-qj5fl\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:37.892469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:37.892172 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:38.203686 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:09:38.203655 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57e55e11_99aa_4bd3_8bb8_3f47a0273f43.slice/crio-1c459557e99e2f4cef131e4145421c4fd445df36807aaf519da2567e7c3ec181 WatchSource:0}: Error finding container 1c459557e99e2f4cef131e4145421c4fd445df36807aaf519da2567e7c3ec181: Status 404 returned error can't find the container with id 1c459557e99e2f4cef131e4145421c4fd445df36807aaf519da2567e7c3ec181 Oct 30 14:09:38.206024 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:38.205994 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl"] Oct 30 14:09:38.353788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:38.353722 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" event={"ID":"57e55e11-99aa-4bd3-8bb8-3f47a0273f43","Type":"ContainerStarted","Data":"82ffaca3073ffbb1db9df06893dc7730931903a22e008804314169602956a3ce"} Oct 30 14:09:38.353788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:38.353779 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" event={"ID":"57e55e11-99aa-4bd3-8bb8-3f47a0273f43","Type":"ContainerStarted","Data":"1c459557e99e2f4cef131e4145421c4fd445df36807aaf519da2567e7c3ec181"} Oct 30 14:09:38.355876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:38.355842 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" event={"ID":"1f8f2355-d43f-4945-986b-0528d04e18cc","Type":"ContainerStarted","Data":"87c842d0d9c4d66624bcc9f4b75bb76f5f2cf7daa80ac68733d42ea931ef2c20"} Oct 30 14:09:38.356010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:38.355902 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" event={"ID":"1f8f2355-d43f-4945-986b-0528d04e18cc","Type":"ContainerStarted","Data":"66c82217f2746fcef2cf9df02e9aa311415e83b6e8f69ed1b83cfc7a827e792d"} Oct 30 14:09:38.405273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:38.405221 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podStartSLOduration=1.405206647 podStartE2EDuration="1.405206647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:38.401708992 +0000 UTC m=+1208.202214436" watchObservedRunningTime="2026-10-30 14:09:38.405206647 +0000 UTC m=+1208.205712130" Oct 30 14:09:38.448415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:38.448361 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" podStartSLOduration=2.448345567 podStartE2EDuration="2.448345567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:38.445811665 +0000 UTC m=+1208.246317109" watchObservedRunningTime="2026-10-30 14:09:38.448345567 +0000 UTC m=+1208.248851017" Oct 30 14:09:38.892837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:38.892758 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:39.395476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:39.395441 5824 generic.go:356] "Generic (PLEG): container finished" podID="1f8f2355-d43f-4945-986b-0528d04e18cc" containerID="87c842d0d9c4d66624bcc9f4b75bb76f5f2cf7daa80ac68733d42ea931ef2c20" exitCode=0 Oct 30 14:09:39.396955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:39.396929 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" event={"ID":"1f8f2355-d43f-4945-986b-0528d04e18cc","Type":"ContainerDied","Data":"87c842d0d9c4d66624bcc9f4b75bb76f5f2cf7daa80ac68733d42ea931ef2c20"} Oct 30 14:09:40.783212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.779616 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:40.831734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.828410 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-xks-ca\") pod \"1f8f2355-d43f-4945-986b-0528d04e18cc\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " Oct 30 14:09:40.831734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.828443 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8f2355-d43f-4945-986b-0528d04e18cc-kube-api-access-4vh6b\" (UniqueName: \"kubernetes.io/projected/1f8f2355-d43f-4945-986b-0528d04e18cc-kube-api-access-4vh6b\") pod \"1f8f2355-d43f-4945-986b-0528d04e18cc\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " Oct 30 14:09:40.831734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.828486 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-openshift-ca\") pod \"1f8f2355-d43f-4945-986b-0528d04e18cc\" (UID: \"1f8f2355-d43f-4945-986b-0528d04e18cc\") " Oct 30 14:09:40.831734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.829442 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-openshift-ca" pod "1f8f2355-d43f-4945-986b-0528d04e18cc" (UID: "1f8f2355-d43f-4945-986b-0528d04e18cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:40.831734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.831097 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-xks-ca" pod "1f8f2355-d43f-4945-986b-0528d04e18cc" (UID: "1f8f2355-d43f-4945-986b-0528d04e18cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:09:40.836513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.833442 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8f2355-d43f-4945-986b-0528d04e18cc-kube-api-access-4vh6b" pod "1f8f2355-d43f-4945-986b-0528d04e18cc" (UID: "1f8f2355-d43f-4945-986b-0528d04e18cc"). InnerVolumeSpecName "kube-api-access-4vh6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:40.931373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.931338 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:40.931373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.931375 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vh6b\" (UniqueName: \"kubernetes.io/projected/1f8f2355-d43f-4945-986b-0528d04e18cc-kube-api-access-4vh6b\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:40.931615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:40.931394 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f2355-d43f-4945-986b-0528d04e18cc-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:41.427923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:41.427889 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" event={"ID":"1f8f2355-d43f-4945-986b-0528d04e18cc","Type":"ContainerDied","Data":"66c82217f2746fcef2cf9df02e9aa311415e83b6e8f69ed1b83cfc7a827e792d"} Oct 30 14:09:41.428082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:41.427947 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66c82217f2746fcef2cf9df02e9aa311415e83b6e8f69ed1b83cfc7a827e792d" Oct 30 14:09:41.428082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:41.427904 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp" Oct 30 14:09:47.892400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:47.892371 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:47.904513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:47.904489 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:48.487393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:48.487361 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:09:50.967790 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:50.967755 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw"] Oct 30 14:09:50.968615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:50.968592 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1f8f2355-d43f-4945-986b-0528d04e18cc" containerName="revoke-keys" Oct 30 14:09:50.968615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:50.968615 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8f2355-d43f-4945-986b-0528d04e18cc" containerName="revoke-keys" Oct 30 14:09:50.968860 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:50.968845 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8f2355-d43f-4945-986b-0528d04e18cc" containerName="revoke-keys" Oct 30 14:09:50.972414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:50.972392 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:50.994028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:50.993997 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw"] Oct 30 14:09:51.003871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.003844 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:09:51.004118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.004094 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:09:51.058008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.057980 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.058136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.058019 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.058250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.058228 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl44v\" (UniqueName: \"kubernetes.io/projected/6ec71eae-a7c6-4012-bbed-eb9cd934564c-kube-api-access-pl44v\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.159446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.159414 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl44v\" (UniqueName: \"kubernetes.io/projected/6ec71eae-a7c6-4012-bbed-eb9cd934564c-kube-api-access-pl44v\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.159571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.159543 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.159571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.159563 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.160165 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.160141 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.161912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.161891 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.185581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.185556 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl44v\" (UniqueName: \"kubernetes.io/projected/6ec71eae-a7c6-4012-bbed-eb9cd934564c-kube-api-access-pl44v\") pod \"maas-api-revoke-sub-48fdb4e5-nmvkw\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.298388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.298328 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:51.604866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:51.604826 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw"] Oct 30 14:09:51.605426 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:09:51.605394 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec71eae_a7c6_4012_bbed_eb9cd934564c.slice/crio-5454b63e2d15adf6fc85d97912148801c6167b115208bd15d61ab5da6521d756 WatchSource:0}: Error finding container 5454b63e2d15adf6fc85d97912148801c6167b115208bd15d61ab5da6521d756: Status 404 returned error can't find the container with id 5454b63e2d15adf6fc85d97912148801c6167b115208bd15d61ab5da6521d756 Oct 30 14:09:52.508361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:52.508329 5824 generic.go:356] "Generic (PLEG): container finished" podID="6ec71eae-a7c6-4012-bbed-eb9cd934564c" containerID="54f1d15d5b958873cd3099117da40a88af3d92913d06601efe9cb7445b648884" exitCode=0 Oct 30 14:09:52.508701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:52.508381 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" event={"ID":"6ec71eae-a7c6-4012-bbed-eb9cd934564c","Type":"ContainerDied","Data":"54f1d15d5b958873cd3099117da40a88af3d92913d06601efe9cb7445b648884"} Oct 30 14:09:52.508701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:52.508425 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" event={"ID":"6ec71eae-a7c6-4012-bbed-eb9cd934564c","Type":"ContainerStarted","Data":"5454b63e2d15adf6fc85d97912148801c6167b115208bd15d61ab5da6521d756"} Oct 30 14:09:54.397240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.384077 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:54.484634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.484251 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:54.508269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.508241 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-openshift-ca\") pod \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " Oct 30 14:09:54.518626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.518555 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-xks-ca\") pod \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " Oct 30 14:09:54.518844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.518810 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec71eae-a7c6-4012-bbed-eb9cd934564c-kube-api-access-pl44v\" (UniqueName: \"kubernetes.io/projected/6ec71eae-a7c6-4012-bbed-eb9cd934564c-kube-api-access-pl44v\") pod \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\" (UID: \"6ec71eae-a7c6-4012-bbed-eb9cd934564c\") " Oct 30 14:09:54.521628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.521587 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:54.540218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.537384 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-openshift-ca" pod "6ec71eae-a7c6-4012-bbed-eb9cd934564c" (UID: "6ec71eae-a7c6-4012-bbed-eb9cd934564c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:09:54.650783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.650742 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:54.692362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.692324 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:09:54.824147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.824053 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" event={"ID":"6ec71eae-a7c6-4012-bbed-eb9cd934564c","Type":"ContainerDied","Data":"5454b63e2d15adf6fc85d97912148801c6167b115208bd15d61ab5da6521d756"} Oct 30 14:09:54.824430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.824408 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5454b63e2d15adf6fc85d97912148801c6167b115208bd15d61ab5da6521d756" Oct 30 14:09:54.824619 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.824604 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw" Oct 30 14:09:54.893499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.892066 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 14:09:54.913125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.913090 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec71eae-a7c6-4012-bbed-eb9cd934564c-kube-api-access-pl44v" pod "6ec71eae-a7c6-4012-bbed-eb9cd934564c" (UID: "6ec71eae-a7c6-4012-bbed-eb9cd934564c"). InnerVolumeSpecName "kube-api-access-pl44v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:09:55.019213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.993256 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 14:09:55.019213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:54.994619 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-xks-ca" pod "6ec71eae-a7c6-4012-bbed-eb9cd934564c" (UID: "6ec71eae-a7c6-4012-bbed-eb9cd934564c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:09:55.019213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.015558 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec71eae-a7c6-4012-bbed-eb9cd934564c-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:55.019213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.015590 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl44v\" (UniqueName: \"kubernetes.io/projected/6ec71eae-a7c6-4012-bbed-eb9cd934564c-kube-api-access-pl44v\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:09:55.019213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.017087 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.148266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.145500 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:09:55.247214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.246296 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.396317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.396270 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:09:55.396903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.396872 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.397316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.397292 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:09:55.412439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.410432 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:09:55.710210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.709386 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:09:55.710210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.710073 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:09:55.723731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.711823 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:09:55.723731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.712104 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:09:55.920299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.919202 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:55.928217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.920592 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:09:55.999265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.999107 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.021213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:55.999688 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:09:56.021213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.003797 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.021213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.019621 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:09:56.021213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.019957 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:09:56.157212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.156473 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:09:56.157212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.157102 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:09:56.223238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.196648 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.253496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.250288 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.253496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.250644 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:09:56.396810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.396768 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.397766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.397735 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 14:09:56.418353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.418320 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.492724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.487360 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.695274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.679803 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:09:56.695274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.679919 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:09:56.695274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.680244 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:09:56.702644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.702221 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:09:56.711599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.710927 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:09:56.717237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.714261 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:09:56.814931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.814891 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:09:56.897247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.897216 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:09:56.911975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.898315 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:09:56.949959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.949875 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:09:56.949959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.949953 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:09:56.954373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.954141 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:09:56.955579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.955558 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:09:56.962021 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:09:56.962001 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:10:00.467339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.467307 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h"] Oct 30 14:10:00.468242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.468219 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6ec71eae-a7c6-4012-bbed-eb9cd934564c" containerName="revoke-keys" Oct 30 14:10:00.468242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.468240 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec71eae-a7c6-4012-bbed-eb9cd934564c" containerName="revoke-keys" Oct 30 14:10:00.468440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.468427 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec71eae-a7c6-4012-bbed-eb9cd934564c" containerName="revoke-keys" Oct 30 14:10:00.473705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.473681 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.487786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.487763 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h"] Oct 30 14:10:00.497709 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.497684 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"openshift-service-ca.crt\"" Oct 30 14:10:00.497824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.497714 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"kube-root-ca.crt\"" Oct 30 14:10:00.507529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.507502 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"default-dockercfg-wfvcl\"" Oct 30 14:10:00.507666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.507645 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2df0ab\"/\"distinct-w0-2df0ab-kserve-self-signed-certs\"" Oct 30 14:10:00.572741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.572696 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2cp8\" (UniqueName: \"kubernetes.io/projected/489d0e4e-50f7-48c7-8df9-7482a3455878-kube-api-access-r2cp8\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.572741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.572735 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-tmp-dir\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.572920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.572766 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489d0e4e-50f7-48c7-8df9-7482a3455878-tls-certs\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.572964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.572947 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-home\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.572999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.572971 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-model-cache\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.573050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.573034 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-dshm\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.674699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.674657 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2cp8\" (UniqueName: \"kubernetes.io/projected/489d0e4e-50f7-48c7-8df9-7482a3455878-kube-api-access-r2cp8\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.675345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.675318 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-tmp-dir\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.675940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.675922 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489d0e4e-50f7-48c7-8df9-7482a3455878-tls-certs\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.679144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.676758 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-home\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.679775 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.679758 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-model-cache\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.680005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.675852 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-tmp-dir\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.680005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.679704 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-home\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.680140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.680059 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-model-cache\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.680312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.680295 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-dshm\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.696126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.696096 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-dshm\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.699722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.699697 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489d0e4e-50f7-48c7-8df9-7482a3455878-tls-certs\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.706083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.706021 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2cp8\" (UniqueName: \"kubernetes.io/projected/489d0e4e-50f7-48c7-8df9-7482a3455878-kube-api-access-r2cp8\") pod \"distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:00.789483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:00.789398 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:01.116462 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:01.116434 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod489d0e4e_50f7_48c7_8df9_7482a3455878.slice/crio-802f5683b59d65d28f4ffc325193b501ea0e01dc60b7a5f30a0aa606150d0c82 WatchSource:0}: Error finding container 802f5683b59d65d28f4ffc325193b501ea0e01dc60b7a5f30a0aa606150d0c82: Status 404 returned error can't find the container with id 802f5683b59d65d28f4ffc325193b501ea0e01dc60b7a5f30a0aa606150d0c82 Oct 30 14:10:01.119178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:01.119152 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h"] Oct 30 14:10:01.988684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:01.988649 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" event={"ID":"489d0e4e-50f7-48c7-8df9-7482a3455878","Type":"ContainerStarted","Data":"1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa"} Oct 30 14:10:01.989074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:01.988695 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" event={"ID":"489d0e4e-50f7-48c7-8df9-7482a3455878","Type":"ContainerStarted","Data":"802f5683b59d65d28f4ffc325193b501ea0e01dc60b7a5f30a0aa606150d0c82"} Oct 30 14:10:02.038760 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:02.038712 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podStartSLOduration=2.038697952 podStartE2EDuration="2.038697952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:02.035832268 +0000 UTC m=+1231.836337712" watchObservedRunningTime="2026-10-30 14:10:02.038697952 +0000 UTC m=+1231.839203413" Oct 30 14:10:02.790129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:02.790094 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:03.641844 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:10:03.641808 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:10:05.466683 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.466651 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv"] Oct 30 14:10:05.471016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.470999 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.489077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.489049 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv"] Oct 30 14:10:05.497714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.497696 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:10:05.497931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.497914 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:10:05.561843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.561811 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44e621e-6b30-4220-b7a3-229158196067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.561965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.561883 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lf4\" (UniqueName: \"kubernetes.io/projected/f44e621e-6b30-4220-b7a3-229158196067-kube-api-access-k2lf4\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.562063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.562043 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44e621e-6b30-4220-b7a3-229158196067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.663777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.663753 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44e621e-6b30-4220-b7a3-229158196067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.663884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.663813 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lf4\" (UniqueName: \"kubernetes.io/projected/f44e621e-6b30-4220-b7a3-229158196067-kube-api-access-k2lf4\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.663959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.663944 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44e621e-6b30-4220-b7a3-229158196067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.664567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.664547 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44e621e-6b30-4220-b7a3-229158196067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.666069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.666049 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44e621e-6b30-4220-b7a3-229158196067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.689708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.689683 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lf4\" (UniqueName: \"kubernetes.io/projected/f44e621e-6b30-4220-b7a3-229158196067-kube-api-access-k2lf4\") pod \"maas-api-revoke-sub-72f02099-ptwrv\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:05.786554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:05.786496 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:06.097817 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:06.097788 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44e621e_6b30_4220_b7a3_229158196067.slice/crio-efa5c8ffa3292dde49e75a27fd145e008e0af801b06bf4d04ab8a35b348ea550 WatchSource:0}: Error finding container efa5c8ffa3292dde49e75a27fd145e008e0af801b06bf4d04ab8a35b348ea550: Status 404 returned error can't find the container with id efa5c8ffa3292dde49e75a27fd145e008e0af801b06bf4d04ab8a35b348ea550 Oct 30 14:10:06.104346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:06.104321 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv"] Oct 30 14:10:07.026539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:07.026511 5824 generic.go:356] "Generic (PLEG): container finished" podID="f44e621e-6b30-4220-b7a3-229158196067" containerID="1582cbcd5cffad297d0dd940bd205ebef557a0b2515fb0eb1b5ff1fa3e5999b9" exitCode=0 Oct 30 14:10:07.026925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:07.026550 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" event={"ID":"f44e621e-6b30-4220-b7a3-229158196067","Type":"ContainerDied","Data":"1582cbcd5cffad297d0dd940bd205ebef557a0b2515fb0eb1b5ff1fa3e5999b9"} Oct 30 14:10:07.026925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:07.026614 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" event={"ID":"f44e621e-6b30-4220-b7a3-229158196067","Type":"ContainerStarted","Data":"efa5c8ffa3292dde49e75a27fd145e008e0af801b06bf4d04ab8a35b348ea550"} Oct 30 14:10:08.207592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.207567 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:08.309026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.308943 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44e621e-6b30-4220-b7a3-229158196067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44e621e-6b30-4220-b7a3-229158196067-maas-api-openshift-ca\") pod \"f44e621e-6b30-4220-b7a3-229158196067\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " Oct 30 14:10:08.309026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.309018 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44e621e-6b30-4220-b7a3-229158196067-kube-api-access-k2lf4\" (UniqueName: \"kubernetes.io/projected/f44e621e-6b30-4220-b7a3-229158196067-kube-api-access-k2lf4\") pod \"f44e621e-6b30-4220-b7a3-229158196067\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " Oct 30 14:10:08.309275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.309245 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f44e621e-6b30-4220-b7a3-229158196067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44e621e-6b30-4220-b7a3-229158196067-maas-api-xks-ca\") pod \"f44e621e-6b30-4220-b7a3-229158196067\" (UID: \"f44e621e-6b30-4220-b7a3-229158196067\") " Oct 30 14:10:08.309385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.309354 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44e621e-6b30-4220-b7a3-229158196067-maas-api-openshift-ca" pod "f44e621e-6b30-4220-b7a3-229158196067" (UID: "f44e621e-6b30-4220-b7a3-229158196067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:10:08.309969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.309948 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44e621e-6b30-4220-b7a3-229158196067-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:10:08.311222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.311177 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44e621e-6b30-4220-b7a3-229158196067-maas-api-xks-ca" pod "f44e621e-6b30-4220-b7a3-229158196067" (UID: "f44e621e-6b30-4220-b7a3-229158196067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:10:08.311633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.311616 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44e621e-6b30-4220-b7a3-229158196067-kube-api-access-k2lf4" pod "f44e621e-6b30-4220-b7a3-229158196067" (UID: "f44e621e-6b30-4220-b7a3-229158196067"). InnerVolumeSpecName "kube-api-access-k2lf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:10:08.411421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.411400 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44e621e-6b30-4220-b7a3-229158196067-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:10:08.411421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:08.411420 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2lf4\" (UniqueName: \"kubernetes.io/projected/f44e621e-6b30-4220-b7a3-229158196067-kube-api-access-k2lf4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:10:09.041630 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.041599 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" Oct 30 14:10:09.041777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.041601 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv" event={"ID":"f44e621e-6b30-4220-b7a3-229158196067","Type":"ContainerDied","Data":"efa5c8ffa3292dde49e75a27fd145e008e0af801b06bf4d04ab8a35b348ea550"} Oct 30 14:10:09.041777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.041724 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efa5c8ffa3292dde49e75a27fd145e008e0af801b06bf4d04ab8a35b348ea550" Oct 30 14:10:09.766490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.766456 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs"] Oct 30 14:10:09.767167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.767151 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f44e621e-6b30-4220-b7a3-229158196067" containerName="revoke-keys" Oct 30 14:10:09.767232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.767168 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44e621e-6b30-4220-b7a3-229158196067" containerName="revoke-keys" Oct 30 14:10:09.767380 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.767364 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44e621e-6b30-4220-b7a3-229158196067" containerName="revoke-keys" Oct 30 14:10:09.771376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.771357 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.781635 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.781614 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb"] Oct 30 14:10:09.785863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.785847 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.798238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.798212 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs"] Oct 30 14:10:09.807976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.807950 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb"] Oct 30 14:10:09.826152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.826125 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv"] Oct 30 14:10:09.836993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.836968 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv"] Oct 30 14:10:09.837282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.837260 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.932444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932412 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.932444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932444 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm94d\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-kube-api-access-fm94d\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.932603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932471 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.932603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932497 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5bqs\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-kube-api-access-t5bqs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.932603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932587 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.932728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932611 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.932788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932734 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.932839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932798 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.932891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932852 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.932891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932881 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.933005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932914 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.933005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.932961 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.933070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933045 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.933105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933075 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.933181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933166 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.933274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933202 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.933274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933225 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.933274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933240 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.933429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933287 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:09.933429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933332 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c747c2-f178-4a1c-a666-634e8bc57fef-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.933518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933446 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.933518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933511 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.933595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933561 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.933634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933605 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkstc\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-kube-api-access-dkstc\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:09.933676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933633 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.933740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933727 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:09.933772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:09.933754 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.034982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.034915 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.034982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.034950 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035064 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.035159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035102 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm94d\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-kube-api-access-fm94d\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.035159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035132 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035163 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5bqs\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-kube-api-access-t5bqs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035240 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035273 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.035373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035318 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035346 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.035625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035375 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035402 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.035625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035435 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.035625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035463 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.035625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035509 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035543 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035570 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035690 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.035977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035723 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.035977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035762 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.035977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035789 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.035977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035828 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.035977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035855 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c747c2-f178-4a1c-a666-634e8bc57fef-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.035977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035890 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.035977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.035973 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.036409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036016 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.036409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036060 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkstc\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-kube-api-access-dkstc\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.036409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036090 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.036409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036123 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-credential-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.036409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036142 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.036409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036392 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.036884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036412 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.036884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036684 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-socket\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.036884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036743 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-data\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.037048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.036996 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c747c2-f178-4a1c-a666-634e8bc57fef-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.037048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.037013 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-certs\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.037358 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.037336 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istiod-ca-cert\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.037628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.037594 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-credential-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.037628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.037620 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-socket\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.037795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.037636 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istiod-ca-cert\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.037853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.037797 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-data\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.037853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.037822 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-envoy\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.037964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.037846 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-certs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.038930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.038900 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.039488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.039464 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.039662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.039643 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-podinfo\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.039757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.039737 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-envoy\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.039826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.039807 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-podinfo\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.061531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.061497 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm94d\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-kube-api-access-fm94d\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.061789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.061772 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5bqs\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-kube-api-access-t5bqs\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.062252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.062232 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.062300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.062233 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-token\") pod \"e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.062395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.062376 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkstc\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-kube-api-access-dkstc\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.074368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.074347 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-token\") pod \"e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.087163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.087143 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:10.115389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.115367 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:10.163232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.161072 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:10.409984 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:10.409947 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c747c2_f178_4a1c_a666_634e8bc57fef.slice/crio-74d4cb26083ba281e0626d03183aff0a6d59635f279e0209f3c582bd1b51f141 WatchSource:0}: Error finding container 74d4cb26083ba281e0626d03183aff0a6d59635f279e0209f3c582bd1b51f141: Status 404 returned error can't find the container with id 74d4cb26083ba281e0626d03183aff0a6d59635f279e0209f3c582bd1b51f141 Oct 30 14:10:10.411855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.411794 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs"] Oct 30 14:10:10.412743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.412711 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.412811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.412777 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.412811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.412804 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.449988 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:10.449951 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a0a566_f79f_4fb4_b5be_9d2862d656a5.slice/crio-b4f15f041cd9e2e1b48ebd887479fd1af2bc2d3d9dccecfac258a690387ed017 WatchSource:0}: Error finding container b4f15f041cd9e2e1b48ebd887479fd1af2bc2d3d9dccecfac258a690387ed017: Status 404 returned error can't find the container with id b4f15f041cd9e2e1b48ebd887479fd1af2bc2d3d9dccecfac258a690387ed017 Oct 30 14:10:10.452991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.452843 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.452991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.452907 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb"] Oct 30 14:10:10.452991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.452911 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.453245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.453019 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.501548 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:10.501516 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2c31c7_c84d_4e01_a1db_53b637f7882d.slice/crio-f2d00277defe51d9d6bbccff6e79d5973d87b22028ce1446b44a933b4814f0fe WatchSource:0}: Error finding container f2d00277defe51d9d6bbccff6e79d5973d87b22028ce1446b44a933b4814f0fe: Status 404 returned error can't find the container with id f2d00277defe51d9d6bbccff6e79d5973d87b22028ce1446b44a933b4814f0fe Oct 30 14:10:10.503447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.503418 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv"] Oct 30 14:10:10.504866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.504826 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.504974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.504912 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.504974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.504961 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:10.523723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.523689 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6"] Oct 30 14:10:10.530256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.530233 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.558897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.558861 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6"] Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655593 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655630 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db815bfe-85a0-48fb-858f-015cceef6ea6-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655688 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655706 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655735 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655756 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655809 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655868 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.656101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.655889 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gtx\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-kube-api-access-k7gtx\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.757337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.757307 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.757552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.757537 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gtx\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-kube-api-access-k7gtx\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.757803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.757786 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.757921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.757910 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db815bfe-85a0-48fb-858f-015cceef6ea6-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.758057 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.758037 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.758161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.758149 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.758295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.758273 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.758451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.758319 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.758451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.758410 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.759139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.758849 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.759139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.758898 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.759616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.759575 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.759712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.759628 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.760811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.760780 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db815bfe-85a0-48fb-858f-015cceef6ea6-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.761372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.761348 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.762481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.762454 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.784443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.784417 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.785089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.785071 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gtx\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-kube-api-access-k7gtx\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:10.789707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.789683 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:10.804748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.804725 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:10.859719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:10.859682 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:11.063290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.063174 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" event={"ID":"55c747c2-f178-4a1c-a666-634e8bc57fef","Type":"ContainerStarted","Data":"c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c"} Oct 30 14:10:11.063290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.063262 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" event={"ID":"55c747c2-f178-4a1c-a666-634e8bc57fef","Type":"ContainerStarted","Data":"74d4cb26083ba281e0626d03183aff0a6d59635f279e0209f3c582bd1b51f141"} Oct 30 14:10:11.066451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.066268 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" event={"ID":"7a2c31c7-c84d-4e01-a1db-53b637f7882d","Type":"ContainerStarted","Data":"0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38"} Oct 30 14:10:11.066451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.066329 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" event={"ID":"7a2c31c7-c84d-4e01-a1db-53b637f7882d","Type":"ContainerStarted","Data":"f2d00277defe51d9d6bbccff6e79d5973d87b22028ce1446b44a933b4814f0fe"} Oct 30 14:10:11.068538 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.068511 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" event={"ID":"81a0a566-f79f-4fb4-b5be-9d2862d656a5","Type":"ContainerStarted","Data":"e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa"} Oct 30 14:10:11.068657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.068559 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" event={"ID":"81a0a566-f79f-4fb4-b5be-9d2862d656a5","Type":"ContainerStarted","Data":"b4f15f041cd9e2e1b48ebd887479fd1af2bc2d3d9dccecfac258a690387ed017"} Oct 30 14:10:11.080879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.080857 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:11.087633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.087615 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:11.088950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.088930 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:11.089028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.088981 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:11.116065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.116043 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:11.117546 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.117521 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:11.117657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.117561 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:11.128500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.128455 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podStartSLOduration=2.128442541 podStartE2EDuration="2.128442541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:11.128009007 +0000 UTC m=+1240.928514452" watchObservedRunningTime="2026-10-30 14:10:11.128442541 +0000 UTC m=+1240.928947985" Oct 30 14:10:11.162130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.162099 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:11.163522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.163499 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:11.163623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.163546 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:11.185838 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.185796 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podStartSLOduration=2.185783466 podStartE2EDuration="2.185783466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:11.182108643 +0000 UTC m=+1240.982614086" watchObservedRunningTime="2026-10-30 14:10:11.185783466 +0000 UTC m=+1240.986288983" Oct 30 14:10:11.207611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.207585 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6"] Oct 30 14:10:11.216998 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:11.216971 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb815bfe_85a0_48fb_858f_015cceef6ea6.slice/crio-0114948bed20187994a77a4f34e43c3255a512bb1949c0c943df6751e696385f WatchSource:0}: Error finding container 0114948bed20187994a77a4f34e43c3255a512bb1949c0c943df6751e696385f: Status 404 returned error can't find the container with id 0114948bed20187994a77a4f34e43c3255a512bb1949c0c943df6751e696385f Oct 30 14:10:11.220227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.220169 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:11.220327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.220278 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:11.220396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.220323 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:11.234231 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:11.234160 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podStartSLOduration=2.234146958 podStartE2EDuration="2.234146958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:11.233269483 +0000 UTC m=+1241.033774927" watchObservedRunningTime="2026-10-30 14:10:11.234146958 +0000 UTC m=+1241.034652402" Oct 30 14:10:12.111923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.111880 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:12.169230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.152138 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:12.231214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.221576 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:12.231214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.221644 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:12.231214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.222636 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:12.231214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.222675 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:12.236835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.236801 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" event={"ID":"db815bfe-85a0-48fb-858f-015cceef6ea6","Type":"ContainerStarted","Data":"e6e4a8820f531de05088f349cde66b8629319c4b91110e0bfe28e549e62c01fb"} Oct 30 14:10:12.237272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.237230 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" event={"ID":"db815bfe-85a0-48fb-858f-015cceef6ea6","Type":"ContainerStarted","Data":"0114948bed20187994a77a4f34e43c3255a512bb1949c0c943df6751e696385f"} Oct 30 14:10:12.339293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.334928 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podStartSLOduration=2.33490886 podStartE2EDuration="2.33490886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:12.33094004 +0000 UTC m=+1242.131445489" watchObservedRunningTime="2026-10-30 14:10:12.33490886 +0000 UTC m=+1242.135414305" Oct 30 14:10:12.867212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.866311 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:12.867212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.866473 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:12.867212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:12.866520 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:13.100214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.099263 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:13.100214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.099324 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:13.125269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.118513 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:13.125269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.118574 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:13.165677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.165637 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:13.165952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.165908 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:13.723214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.722531 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:13.774214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.766289 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:13.824389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.824287 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:13.876593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.872674 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:13.876593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.872747 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:13.907337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.906274 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:13.980279 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.972296 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:13.980279 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.972692 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:10:13.980279 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.972970 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:10:13.980279 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:13.973241 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:10:14.077215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.076272 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 14:10:14.102214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.094825 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:14.102214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.094888 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:14.120211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.110276 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:14.131211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.120571 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:14.131211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.120623 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:14.173212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.173074 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:14.173212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.173136 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:14.314370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.314268 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:14.490360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.490319 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:14.741217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.735276 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:14.741217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.736051 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:10:14.773344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.773278 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:14.774003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.773977 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:10:14.849421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.849379 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 14:10:15.159673 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.876643 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:15.370617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.370572 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:15.371032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.925353 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.016303 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.016357 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.016384 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.016410 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.128203 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.167311 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.373423 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.373469 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.373508 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.373544 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.373582 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.373619 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:14.925315 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.277361 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.374778 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.277416 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.374829 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.280776 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:15.379346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.374870 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:15.385268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.385229 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.424264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.394582 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:10:15.523214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.503133 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.585258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.549484 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:10:15.758270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.736547 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:15.995866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.995825 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:15.996134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:15.996101 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:16.005515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.005471 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.021949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.021832 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.132650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.131286 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:16.132650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.131348 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:16.164368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.164328 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:10:16.210286 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.184471 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.318665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.318564 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.462106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.460597 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.527308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.477375 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.527308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.482105 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:10:16.527308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.499232 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:16.527308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.499298 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:16.527308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.523913 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:16.527308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.523982 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:16.561211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.545525 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.577272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.558342 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:10:16.602214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.558743 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.788222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.776537 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:16.847072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.842899 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 14:10:16.877654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:16.877603 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:10:17.018214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.013269 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:10:17.018214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.013325 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.090238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.075350 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:17.090238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.075408 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:17.166271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.147087 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:17.166271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.147150 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:17.172253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.172218 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:17.187348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.172433 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:17.213453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.188883 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:17.335654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.335611 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:17.335918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.218875 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:10:17.345308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.345263 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.500220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.499321 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.526216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.516565 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.585519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.585485 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:10:17.585943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.585354 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:10:17.586183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.586157 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.790269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.783680 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:17.908243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.907394 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 14:10:17.924869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.924830 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:17.925139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:17.925105 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:18.051268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.002207 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.051268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.043101 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.051268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.043347 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.204327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.190899 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:18.204327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.190960 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:18.204327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.191052 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:18.204327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.191079 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:18.209212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.205072 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:18.209212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.205127 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:18.268350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.239969 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.400601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.370314 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.400601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.370661 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:10:18.400601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.370992 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.511212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.505558 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:10:18.586214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.525953 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.793231 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.586606 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:10:18.801906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.617662 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:10:18.840217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.652431 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:10:18.842162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.785212 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.938400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.938344 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.938986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.938965 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:10:18.986238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.977389 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:10:18.986238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.977456 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:18.986672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.986635 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:18.986863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:18.986825 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:19.016506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.016466 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:10:19.036780 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.036742 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.050736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.050640 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.077247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.071596 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 14:10:19.143305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.143255 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:19.143678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.143636 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:19.144084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.144063 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:19.144261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.144237 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:19.164272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.164244 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:19.164496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.164474 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:19.390267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.382277 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.390267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.382997 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 14:10:19.426829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.426780 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:10:19.427145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.427098 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.595944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.595757 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.696416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.696345 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.803682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.803635 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.804303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.804273 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:10:19.895414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.833330 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.895414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.864823 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:19.895414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.866745 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:19.895414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.866796 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:19.895414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:19.866863 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.031287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.028004 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:10:20.066909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.064129 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.080783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.080732 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.081392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.081345 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.278257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.278213 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:20.278589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.278555 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:20.279276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.279248 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 14:10:20.279469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.279446 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 14:10:20.387706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.382436 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:20.387706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.382516 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:20.388458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.388414 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.414216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.389336 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.638297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.542322 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:10:20.809426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.809369 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:10:20.832165 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.809938 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.859244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.837604 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.865110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.865063 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:20.878223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.874762 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:20.878223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.874820 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:20.878223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:20.874896 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.059214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.048328 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.075478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.075333 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.084245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.083798 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.086918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.084535 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.087690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.087586 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 14:10:21.087690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.087631 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 14:10:21.259806 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.259769 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 14:10:21.260051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.260018 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 14:10:21.410269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.409009 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.425388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.424495 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.587707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.587401 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:10:21.649921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.649886 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:10:21.699675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.699652 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:10:21.715663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.715638 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:10:21.853230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.853156 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.886363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.886311 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:21.896174 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.896150 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:10:21.986280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.985735 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 14:10:21.986280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:21.985795 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 14:10:22.040645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.013421 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 14:10:22.090214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.070120 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:10:22.092791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.092767 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:10:22.105126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.105083 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:22.162805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.162772 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:10:22.163119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.163087 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:22.467446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.467399 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:10:22.467912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.467480 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:22.517988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.517943 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:22.592100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.592066 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:10:22.592405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.592384 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:10:22.641213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.638038 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:10:22.712686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.707483 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:22.713482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.713235 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:22.715350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.713994 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:10:22.715531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.715500 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:22.716984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.716964 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:10:22.737246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.737157 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:22.742943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.742734 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:22.759363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.759337 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:10:22.763814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.763792 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:10:22.791913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.791892 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:10:22.883534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:22.883508 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:10:23.098255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:23.094233 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:23.098255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:23.095251 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:23.098255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:23.096125 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:10:23.316277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:23.316246 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:23.317298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:23.317276 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:23.319305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:23.319266 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:10:27.185606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.185576 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft"] Oct 30 14:10:27.190097 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.190078 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.212322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.212297 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft"] Oct 30 14:10:27.220255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.220230 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 14:10:27.220367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.220252 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 14:10:27.220367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.220283 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 14:10:27.223648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.223625 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zzhpw\"" Oct 30 14:10:27.256498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.256470 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.256637 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.256526 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.256721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.256682 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.256784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.256746 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.256843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.256835 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd63488-9f69-4048-ae51-e8c9d1df74cb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.256901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.256857 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69ct\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-kube-api-access-x69ct\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.256966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.256934 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.358001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.357973 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.358122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.358074 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.358167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.358141 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd63488-9f69-4048-ae51-e8c9d1df74cb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.358259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.358177 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69ct\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-kube-api-access-x69ct\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.358259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.358249 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.358377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.358346 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.358657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.358457 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.359141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.359113 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.360572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.360543 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.360663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.360635 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd63488-9f69-4048-ae51-e8c9d1df74cb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.360704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.360672 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.360782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.360764 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.384358 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.384331 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.384567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.384545 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69ct\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-kube-api-access-x69ct\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4bft\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.504024 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.503969 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:27.826334 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:27.826304 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd63488_9f69_4048_ae51_e8c9d1df74cb.slice/crio-2b506c1011d921f5bb2398da04fa461ced0ed14ca533a3a8fa52f02f011e0a4e WatchSource:0}: Error finding container 2b506c1011d921f5bb2398da04fa461ced0ed14ca533a3a8fa52f02f011e0a4e: Status 404 returned error can't find the container with id 2b506c1011d921f5bb2398da04fa461ced0ed14ca533a3a8fa52f02f011e0a4e Oct 30 14:10:27.827547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.827522 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft"] Oct 30 14:10:27.828540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:27.828525 5824 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:10:28.601570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:28.601533 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" event={"ID":"ddd63488-9f69-4048-ae51-e8c9d1df74cb","Type":"ContainerStarted","Data":"2b506c1011d921f5bb2398da04fa461ced0ed14ca533a3a8fa52f02f011e0a4e"} Oct 30 14:10:30.444699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:30.444660 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:30.445115 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:30.444732 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:30.622772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:30.622734 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" event={"ID":"ddd63488-9f69-4048-ae51-e8c9d1df74cb","Type":"ContainerStarted","Data":"5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8"} Oct 30 14:10:30.622943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:30.622790 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:30.710678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:30.710574 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" podStartSLOduration=1.095653677 podStartE2EDuration="3.710561901s" totalImagesPullingTime="2.614908224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:30.69404061 +0000 UTC m=+1260.494546054" watchObservedRunningTime="2026-10-30 14:10:30.710561901 +0000 UTC m=+1260.511067346" Oct 30 14:10:31.636625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:31.636319 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:10:33.647821 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:10:33.647778 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:10:37.198222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:37.196769 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:37.389214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:37.388205 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 14:10:37.436938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:37.436310 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:10:39.636247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.636210 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm"] Oct 30 14:10:39.642844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.642822 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.658140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.658113 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm"] Oct 30 14:10:39.666153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.666130 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:10:39.666367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.666132 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:10:39.760913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.760879 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.761119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.761103 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgwn\" (UniqueName: \"kubernetes.io/projected/6331517a-78ea-444f-a92c-219a726b9bbb-kube-api-access-ltgwn\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.761320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.761296 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.863585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.863549 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgwn\" (UniqueName: \"kubernetes.io/projected/6331517a-78ea-444f-a92c-219a726b9bbb-kube-api-access-ltgwn\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.863766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.863706 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.863831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.863799 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.864477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.864446 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.866689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.866664 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.889751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.889699 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgwn\" (UniqueName: \"kubernetes.io/projected/6331517a-78ea-444f-a92c-219a726b9bbb-kube-api-access-ltgwn\") pod \"maas-api-revoke-sub-7bc95d2e-bnvfm\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:39.965959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:39.965934 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:40.278047 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:40.278018 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6331517a_78ea_444f_a92c_219a726b9bbb.slice/crio-de69c52f49f0347e13145eaaff95823528eb474a122373e3c0d82ec0f7ed5c13 WatchSource:0}: Error finding container de69c52f49f0347e13145eaaff95823528eb474a122373e3c0d82ec0f7ed5c13: Status 404 returned error can't find the container with id de69c52f49f0347e13145eaaff95823528eb474a122373e3c0d82ec0f7ed5c13 Oct 30 14:10:40.280954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:40.280928 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm"] Oct 30 14:10:40.997346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:40.997263 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" event={"ID":"6331517a-78ea-444f-a92c-219a726b9bbb","Type":"ContainerStarted","Data":"ebdf343821e36bfd899609add151bc63c38468390252a3e9d005c6e57c37e269"} Oct 30 14:10:40.997710 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:40.997359 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" event={"ID":"6331517a-78ea-444f-a92c-219a726b9bbb","Type":"ContainerStarted","Data":"de69c52f49f0347e13145eaaff95823528eb474a122373e3c0d82ec0f7ed5c13"} Oct 30 14:10:41.048417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:41.048372 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" podStartSLOduration=2.048358733 podStartE2EDuration="2.048358733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:41.04662137 +0000 UTC m=+1270.847126818" watchObservedRunningTime="2026-10-30 14:10:41.048358733 +0000 UTC m=+1270.848864177" Oct 30 14:10:42.005845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:42.005805 5824 generic.go:356] "Generic (PLEG): container finished" podID="6331517a-78ea-444f-a92c-219a726b9bbb" containerID="ebdf343821e36bfd899609add151bc63c38468390252a3e9d005c6e57c37e269" exitCode=0 Oct 30 14:10:42.006318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:42.005847 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" event={"ID":"6331517a-78ea-444f-a92c-219a726b9bbb","Type":"ContainerDied","Data":"ebdf343821e36bfd899609add151bc63c38468390252a3e9d005c6e57c37e269"} Oct 30 14:10:43.210244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.210216 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:43.319009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.318941 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-xks-ca\") pod \"6331517a-78ea-444f-a92c-219a726b9bbb\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " Oct 30 14:10:43.319009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.318992 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-openshift-ca\") pod \"6331517a-78ea-444f-a92c-219a726b9bbb\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " Oct 30 14:10:43.319171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.319142 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6331517a-78ea-444f-a92c-219a726b9bbb-kube-api-access-ltgwn\" (UniqueName: \"kubernetes.io/projected/6331517a-78ea-444f-a92c-219a726b9bbb-kube-api-access-ltgwn\") pod \"6331517a-78ea-444f-a92c-219a726b9bbb\" (UID: \"6331517a-78ea-444f-a92c-219a726b9bbb\") " Oct 30 14:10:43.319394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.319365 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-openshift-ca" pod "6331517a-78ea-444f-a92c-219a726b9bbb" (UID: "6331517a-78ea-444f-a92c-219a726b9bbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:10:43.319901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.319879 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:10:43.321641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.321615 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6331517a-78ea-444f-a92c-219a726b9bbb-kube-api-access-ltgwn" pod "6331517a-78ea-444f-a92c-219a726b9bbb" (UID: "6331517a-78ea-444f-a92c-219a726b9bbb"). InnerVolumeSpecName "kube-api-access-ltgwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:10:43.321742 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.321638 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-xks-ca" pod "6331517a-78ea-444f-a92c-219a726b9bbb" (UID: "6331517a-78ea-444f-a92c-219a726b9bbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:10:43.421973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.421937 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltgwn\" (UniqueName: \"kubernetes.io/projected/6331517a-78ea-444f-a92c-219a726b9bbb-kube-api-access-ltgwn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:10:43.421973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:43.421971 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6331517a-78ea-444f-a92c-219a726b9bbb-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:10:44.024448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:44.024419 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" Oct 30 14:10:44.024600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:44.024452 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm" event={"ID":"6331517a-78ea-444f-a92c-219a726b9bbb","Type":"ContainerDied","Data":"de69c52f49f0347e13145eaaff95823528eb474a122373e3c0d82ec0f7ed5c13"} Oct 30 14:10:44.024600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:44.024509 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de69c52f49f0347e13145eaaff95823528eb474a122373e3c0d82ec0f7ed5c13" Oct 30 14:10:55.698944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.698913 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff"] Oct 30 14:10:55.699847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.699816 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6331517a-78ea-444f-a92c-219a726b9bbb" containerName="revoke-keys" Oct 30 14:10:55.699847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.699838 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6331517a-78ea-444f-a92c-219a726b9bbb" containerName="revoke-keys" Oct 30 14:10:55.700025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.700017 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6331517a-78ea-444f-a92c-219a726b9bbb" containerName="revoke-keys" Oct 30 14:10:55.707465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.707443 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.749108 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.749077 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff"] Oct 30 14:10:55.801094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.801044 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.801389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.801367 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.801616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.801599 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.801766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.801749 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.801872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.801858 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3798ac-47a0-4fae-9b07-483c1ead2451-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.801970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.801957 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.802077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.802064 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.802583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.802547 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.802680 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.802611 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvlb\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-kube-api-access-5pvlb\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904097 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvlb\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-kube-api-access-5pvlb\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904225 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904266 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904351 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904401 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904419 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3798ac-47a0-4fae-9b07-483c1ead2451-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904444 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904484 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904671 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.904999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.904971 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-certs\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.905091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.905004 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.905260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.905241 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-credential-socket\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.905358 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.905331 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3798ac-47a0-4fae-9b07-483c1ead2451-istiod-ca-cert\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.905463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.905440 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-data\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.907017 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.906990 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-podinfo\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.907309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.907289 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-envoy\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.930621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.930591 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-token\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:55.930732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:55.930704 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvlb\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-kube-api-access-5pvlb\") pod \"e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:56.044815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:56.044740 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:56.381641 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:10:56.381614 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3798ac_47a0_4fae_9b07_483c1ead2451.slice/crio-36de1f4ad8d862d4c67575cbbacd6c24b410ad92ee75ce197e5e316ce74d925e WatchSource:0}: Error finding container 36de1f4ad8d862d4c67575cbbacd6c24b410ad92ee75ce197e5e316ce74d925e: Status 404 returned error can't find the container with id 36de1f4ad8d862d4c67575cbbacd6c24b410ad92ee75ce197e5e316ce74d925e Oct 30 14:10:56.383227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:56.383184 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff"] Oct 30 14:10:56.384605 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:56.384565 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:56.384697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:56.384646 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:56.384697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:56.384687 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:10:57.155350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:57.150134 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" event={"ID":"ef3798ac-47a0-4fae-9b07-483c1ead2451","Type":"ContainerStarted","Data":"a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269"} Oct 30 14:10:57.155350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:57.150227 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" event={"ID":"ef3798ac-47a0-4fae-9b07-483c1ead2451","Type":"ContainerStarted","Data":"36de1f4ad8d862d4c67575cbbacd6c24b410ad92ee75ce197e5e316ce74d925e"} Oct 30 14:10:57.221127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:57.221061 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podStartSLOduration=2.221041787 podStartE2EDuration="2.221041787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:57.216563286 +0000 UTC m=+1287.017068730" watchObservedRunningTime="2026-10-30 14:10:57.221041787 +0000 UTC m=+1287.021547231" Oct 30 14:10:58.069020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:58.068468 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:10:58.069020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:58.068644 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:10:58.069020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:58.068693 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:10:59.056332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:59.055525 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:10:59.056332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:10:59.055585 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:00.084212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:00.073980 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:11:00.084212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:00.074038 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:01.046399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:01.046356 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:11:01.047226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:01.047169 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:01.588931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:01.588883 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:01.785213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:01.784271 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:01.794211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:01.791814 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Oct 30 14:11:01.868660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:01.868294 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:01.991534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:01.990296 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.016331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.016278 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.016979 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.016946 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 14:11:02.050223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.049971 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.051216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.051053 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:11:02.051216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.051102 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:02.172380 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.172266 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:11:02.384032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.383968 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.384682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.384647 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:11:02.436400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.436358 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.479719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.479675 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.588751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.588703 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.595310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.595269 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.596260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.595826 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:02.596760 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.596740 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:11:02.796271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.790282 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:11:02.796271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:02.791110 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:11:03.015215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.001303 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.016025 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.030103 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.033062 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.033887 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.033951 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.050963 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.051039 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.052027 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.077523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.064945 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:11:03.172817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.172770 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:11:03.189213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.189169 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:11:03.433011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.432970 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.442247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.433563 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.515139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.475058 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:11:03.515139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.475129 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:11:03.515139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.503585 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:03.515139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.503732 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.515139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.503885 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.515139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.505256 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.515139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.505668 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:11:03.657163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.657115 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.702273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.684176 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:11:03.735218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.695530 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:11:03.750517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.750470 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:03.756414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.756385 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:11:03.803656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:03.803602 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.026430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.026332 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.253493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.253434 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 14:11:04.276949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.276851 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 14:11:04.277510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.277474 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:11:04.301117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.278132 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.301117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.278374 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.301117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.278418 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.473214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.461017 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:11:04.473214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.461098 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:04.551267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.516095 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.551267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.516178 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.551267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.548460 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.576215 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:11:04.571832 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:11:04.647217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.603157 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:11:04.751216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.695600 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:11:04.782748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.782701 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:11:04.797218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.783450 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.997120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.735522 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.997120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.799681 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.997120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.997021 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.997120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.813243 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:04.997120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.997088 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:04.997588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:04.812403 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.117098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.113726 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:05.117098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.113789 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.206302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.206259 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:11:05.351539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.222344 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:11:05.399815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.379910 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:05.399815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.241876 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.399815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.245946 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.399815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.326292 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.399815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.326330 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 14:11:05.562215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.560799 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 14:11:05.739356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.739309 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.785329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.785284 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.786660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.786628 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.997234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.997112 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:05.997815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.997785 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 14:11:05.999082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:05.999052 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.004470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.004394 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.050614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.050577 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:11:06.050871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.050841 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:06.145213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.126473 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:06.145213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.126538 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.224561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.224518 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 14:11:06.364783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.360932 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:11:06.392253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.383368 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.392253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.383926 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:11:06.392253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.384205 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.392253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.384477 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:11:06.433375 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.433333 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:11:06.570826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.570787 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:11:06.573214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.572934 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:11:06.574134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.574100 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:06.583040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.583013 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:11:06.593886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.593857 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:11:06.748221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.747250 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:11:06.748523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.748504 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:11:06.760824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.760803 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:11:06.778208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.775905 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:11:06.778208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.776590 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:11:06.781044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.779735 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:11:06.781044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.780354 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:11:06.792205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:06.789769 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:11:07.045694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:07.045610 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 14:11:07.045694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:07.045667 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 14:11:08.401486 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:08.401444 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:11:08.402060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:08.402037 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:11:08.402895 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:08.402873 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:11:12.440988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.440947 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs"] Oct 30 14:11:12.447004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.446985 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.464851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.464808 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs"] Oct 30 14:11:12.471969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.471910 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:11:12.471969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.471926 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:11:12.562572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.562545 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tjs\" (UniqueName: \"kubernetes.io/projected/427455e4-4f52-48a8-a83b-769a7fd6ee86-kube-api-access-d4tjs\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.562761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.562595 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.562885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.562860 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.669447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.669402 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.669638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.669616 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tjs\" (UniqueName: \"kubernetes.io/projected/427455e4-4f52-48a8-a83b-769a7fd6ee86-kube-api-access-d4tjs\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.669878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.669842 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.670542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.670517 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.672084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.672058 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.695976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.695900 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tjs\" (UniqueName: \"kubernetes.io/projected/427455e4-4f52-48a8-a83b-769a7fd6ee86-kube-api-access-d4tjs\") pod \"maas-api-revoke-sub-d7c83787-7fncs\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:12.766241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:12.766184 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:13.080262 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:11:13.080234 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427455e4_4f52_48a8_a83b_769a7fd6ee86.slice/crio-915c7faae7448d72eedca0badcd99e01ebaffc8415602181ab89e75f9646ac05 WatchSource:0}: Error finding container 915c7faae7448d72eedca0badcd99e01ebaffc8415602181ab89e75f9646ac05: Status 404 returned error can't find the container with id 915c7faae7448d72eedca0badcd99e01ebaffc8415602181ab89e75f9646ac05 Oct 30 14:11:13.082505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:13.082484 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs"] Oct 30 14:11:13.627240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:13.627129 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" event={"ID":"427455e4-4f52-48a8-a83b-769a7fd6ee86","Type":"ContainerStarted","Data":"9c8f06fa5e0000fca6cdb02494e49c81b412b431ba1a8dd33eba70a96b587d33"} Oct 30 14:11:13.627240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:13.627218 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" event={"ID":"427455e4-4f52-48a8-a83b-769a7fd6ee86","Type":"ContainerStarted","Data":"915c7faae7448d72eedca0badcd99e01ebaffc8415602181ab89e75f9646ac05"} Oct 30 14:11:13.674595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:13.674545 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" podStartSLOduration=1.674532273 podStartE2EDuration="1.674532273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:13.671588712 +0000 UTC m=+1303.472094156" watchObservedRunningTime="2026-10-30 14:11:13.674532273 +0000 UTC m=+1303.475037717" Oct 30 14:11:14.636942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:14.636907 5824 generic.go:356] "Generic (PLEG): container finished" podID="427455e4-4f52-48a8-a83b-769a7fd6ee86" containerID="9c8f06fa5e0000fca6cdb02494e49c81b412b431ba1a8dd33eba70a96b587d33" exitCode=0 Oct 30 14:11:14.637366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:14.636957 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" event={"ID":"427455e4-4f52-48a8-a83b-769a7fd6ee86","Type":"ContainerDied","Data":"9c8f06fa5e0000fca6cdb02494e49c81b412b431ba1a8dd33eba70a96b587d33"} Oct 30 14:11:15.838540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:15.838518 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:15.920965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:15.920889 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-xks-ca\") pod \"427455e4-4f52-48a8-a83b-769a7fd6ee86\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " Oct 30 14:11:15.921102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:15.921050 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427455e4-4f52-48a8-a83b-769a7fd6ee86-kube-api-access-d4tjs\" (UniqueName: \"kubernetes.io/projected/427455e4-4f52-48a8-a83b-769a7fd6ee86-kube-api-access-d4tjs\") pod \"427455e4-4f52-48a8-a83b-769a7fd6ee86\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " Oct 30 14:11:15.921166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:15.921101 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-openshift-ca\") pod \"427455e4-4f52-48a8-a83b-769a7fd6ee86\" (UID: \"427455e4-4f52-48a8-a83b-769a7fd6ee86\") " Oct 30 14:11:15.921544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:15.921515 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-openshift-ca" pod "427455e4-4f52-48a8-a83b-769a7fd6ee86" (UID: "427455e4-4f52-48a8-a83b-769a7fd6ee86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:15.921723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:15.921703 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:15.923568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:15.923546 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-xks-ca" pod "427455e4-4f52-48a8-a83b-769a7fd6ee86" (UID: "427455e4-4f52-48a8-a83b-769a7fd6ee86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:15.923641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:15.923572 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427455e4-4f52-48a8-a83b-769a7fd6ee86-kube-api-access-d4tjs" pod "427455e4-4f52-48a8-a83b-769a7fd6ee86" (UID: "427455e4-4f52-48a8-a83b-769a7fd6ee86"). InnerVolumeSpecName "kube-api-access-d4tjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:16.023230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:16.023199 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/427455e4-4f52-48a8-a83b-769a7fd6ee86-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:16.023230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:16.023231 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4tjs\" (UniqueName: \"kubernetes.io/projected/427455e4-4f52-48a8-a83b-769a7fd6ee86-kube-api-access-d4tjs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:16.654300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:16.654217 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" Oct 30 14:11:16.654440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:16.654219 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs" event={"ID":"427455e4-4f52-48a8-a83b-769a7fd6ee86","Type":"ContainerDied","Data":"915c7faae7448d72eedca0badcd99e01ebaffc8415602181ab89e75f9646ac05"} Oct 30 14:11:16.654440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:16.654347 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915c7faae7448d72eedca0badcd99e01ebaffc8415602181ab89e75f9646ac05" Oct 30 14:11:21.416570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.416537 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv"] Oct 30 14:11:21.417466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.417440 5824 cpu_manager.go:414] "containerMap: removing container" podUID="427455e4-4f52-48a8-a83b-769a7fd6ee86" containerName="revoke-keys" Oct 30 14:11:21.417555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.417467 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427455e4-4f52-48a8-a83b-769a7fd6ee86" containerName="revoke-keys" Oct 30 14:11:21.417663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.417649 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="427455e4-4f52-48a8-a83b-769a7fd6ee86" containerName="revoke-keys" Oct 30 14:11:21.421646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.421626 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.439640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.439616 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv"] Oct 30 14:11:21.448154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.448130 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:11:21.448280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.448181 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:11:21.498357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.498331 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qxk\" (UniqueName: \"kubernetes.io/projected/a70eaab3-231a-406f-850e-a8184dd7a0d4-kube-api-access-26qxk\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.498479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.498414 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.498565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.498539 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.599869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.599843 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qxk\" (UniqueName: \"kubernetes.io/projected/a70eaab3-231a-406f-850e-a8184dd7a0d4-kube-api-access-26qxk\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.599957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.599911 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.599957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.599942 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.600577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.600549 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.602305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.602287 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.625952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.625930 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qxk\" (UniqueName: \"kubernetes.io/projected/a70eaab3-231a-406f-850e-a8184dd7a0d4-kube-api-access-26qxk\") pod \"maas-api-revoke-sub-02dd168a-hvpvv\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:21.736597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:21.736564 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:22.055521 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:11:22.055492 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70eaab3_231a_406f_850e_a8184dd7a0d4.slice/crio-b0089a7b4f8ac0539fee83ac5ecefc8e6f1d47885fb16e1e3f5cb707a9112d4f WatchSource:0}: Error finding container b0089a7b4f8ac0539fee83ac5ecefc8e6f1d47885fb16e1e3f5cb707a9112d4f: Status 404 returned error can't find the container with id b0089a7b4f8ac0539fee83ac5ecefc8e6f1d47885fb16e1e3f5cb707a9112d4f Oct 30 14:11:22.058032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:22.058006 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv"] Oct 30 14:11:22.708889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:22.708854 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" event={"ID":"a70eaab3-231a-406f-850e-a8184dd7a0d4","Type":"ContainerStarted","Data":"4c49634f1813de328b1d342b042a43d741656c3a6c75d7ecc5170a49d77c0d10"} Oct 30 14:11:22.709366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:22.708912 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" event={"ID":"a70eaab3-231a-406f-850e-a8184dd7a0d4","Type":"ContainerStarted","Data":"b0089a7b4f8ac0539fee83ac5ecefc8e6f1d47885fb16e1e3f5cb707a9112d4f"} Oct 30 14:11:22.758098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:22.758054 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" podStartSLOduration=1.758039856 podStartE2EDuration="1.758039856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:22.755619861 +0000 UTC m=+1312.556125306" watchObservedRunningTime="2026-10-30 14:11:22.758039856 +0000 UTC m=+1312.558545300" Oct 30 14:11:23.719988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:23.719948 5824 generic.go:356] "Generic (PLEG): container finished" podID="a70eaab3-231a-406f-850e-a8184dd7a0d4" containerID="4c49634f1813de328b1d342b042a43d741656c3a6c75d7ecc5170a49d77c0d10" exitCode=0 Oct 30 14:11:23.720437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:23.720040 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" event={"ID":"a70eaab3-231a-406f-850e-a8184dd7a0d4","Type":"ContainerDied","Data":"4c49634f1813de328b1d342b042a43d741656c3a6c75d7ecc5170a49d77c0d10"} Oct 30 14:11:24.907931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:24.907906 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:25.046054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.045976 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-openshift-ca\") pod \"a70eaab3-231a-406f-850e-a8184dd7a0d4\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " Oct 30 14:11:25.046054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.046033 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70eaab3-231a-406f-850e-a8184dd7a0d4-kube-api-access-26qxk\" (UniqueName: \"kubernetes.io/projected/a70eaab3-231a-406f-850e-a8184dd7a0d4-kube-api-access-26qxk\") pod \"a70eaab3-231a-406f-850e-a8184dd7a0d4\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " Oct 30 14:11:25.046350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.046328 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-xks-ca\") pod \"a70eaab3-231a-406f-850e-a8184dd7a0d4\" (UID: \"a70eaab3-231a-406f-850e-a8184dd7a0d4\") " Oct 30 14:11:25.046403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.046380 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-openshift-ca" pod "a70eaab3-231a-406f-850e-a8184dd7a0d4" (UID: "a70eaab3-231a-406f-850e-a8184dd7a0d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:25.046906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.046875 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:25.048257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.048231 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70eaab3-231a-406f-850e-a8184dd7a0d4-kube-api-access-26qxk" pod "a70eaab3-231a-406f-850e-a8184dd7a0d4" (UID: "a70eaab3-231a-406f-850e-a8184dd7a0d4"). InnerVolumeSpecName "kube-api-access-26qxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:25.048391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.048369 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-xks-ca" pod "a70eaab3-231a-406f-850e-a8184dd7a0d4" (UID: "a70eaab3-231a-406f-850e-a8184dd7a0d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:25.148759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.148705 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26qxk\" (UniqueName: \"kubernetes.io/projected/a70eaab3-231a-406f-850e-a8184dd7a0d4-kube-api-access-26qxk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:25.148759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.148761 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70eaab3-231a-406f-850e-a8184dd7a0d4-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:25.735514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.735491 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" Oct 30 14:11:25.735677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.735530 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv" event={"ID":"a70eaab3-231a-406f-850e-a8184dd7a0d4","Type":"ContainerDied","Data":"b0089a7b4f8ac0539fee83ac5ecefc8e6f1d47885fb16e1e3f5cb707a9112d4f"} Oct 30 14:11:25.735677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:25.735586 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0089a7b4f8ac0539fee83ac5ecefc8e6f1d47885fb16e1e3f5cb707a9112d4f" Oct 30 14:11:29.565319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:29.565271 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 14:11:29.593531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:29.593475 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:29.671366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:29.671303 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:29.746351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:29.746301 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:29.764318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:29.764271 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:29.794841 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:29.794265 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:11:29.794841 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:29.794423 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:29.998343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:29.998289 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:11:30.012694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.012649 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 14:11:30.013337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.013306 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.029231 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.028306 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.229389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.229337 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.286270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.285217 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 14:11:30.369216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.368999 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:11:30.575277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.574725 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.603322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.576099 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:11:30.748265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.746702 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:11:30.748265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.747513 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:11:30.805241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.777980 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:30.805241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.778654 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:11:30.886268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.845518 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:11:30.886268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.845873 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:11:30.886268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.846521 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:11:30.886268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:30.846581 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:11:31.072274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.056299 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.072274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.056660 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:11:31.072274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.067416 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.128431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.076629 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:11:31.128431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.076697 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:11:31.128431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.076733 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:11:31.128431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.095738 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:11:31.128431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.096036 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:11:31.286086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.258081 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:11:31.327216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.327044 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:11:31.435333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.435288 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:11:31.573697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.501255 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:11:31.573697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.501610 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.573697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.502032 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:11:31.595242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.576317 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:31.957360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:31.845502 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.071226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.070691 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:11:32.157259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.125272 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.157259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.126500 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 14:11:32.157259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.129260 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:11:32.157259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.148491 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Oct 30 14:11:32.157259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.151092 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Oct 30 14:11:32.188832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.161040 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:11:32.188832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.165282 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:11:32.196895 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.195423 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:11:32.275556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.259115 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.465562 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.451372 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.503178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.503132 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Oct 30 14:11:32.553938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.503438 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.577664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.577610 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:32.875272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:32.862242 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.073543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.073494 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.157319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.139497 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 14:11:33.157319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.139576 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.157319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.140906 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.161213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.158337 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:33.161213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.158409 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.174468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.174427 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:33.377214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.373312 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.537282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.189440 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:11:33.563744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.449886 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:11:33.686212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.189312 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.733381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.577632 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:11:33.733649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.580769 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.733790 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.581157 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:11:33.733899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.584420 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.734025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.671463 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:33.734183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.734149 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.734330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.698715 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:33.768273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.768232 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:33.768545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:33.768513 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.038213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.012349 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:11:34.074350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.074302 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.153122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.153085 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.154035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.152665 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.281250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.281206 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 14:11:34.281632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.281591 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 14:11:34.332850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.262170 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:11:34.576213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.573324 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:34.576213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.573401 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.576213 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:11:34.573677 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:11:34.635266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.633461 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:11:34.635266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.633520 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.660655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.660617 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:11:34.738871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.738838 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:11:34.782210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.782164 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:11:34.788798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.788777 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:11:34.792494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.792476 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:11:34.815238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.815183 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:11:34.828273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.820584 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:11:34.851713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.851690 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:11:34.863618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.863588 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:34.874572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.874555 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:11:34.892519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.891573 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:11:34.896847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.896829 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:11:34.908942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:34.908926 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:11:38.731708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.731678 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db"] Oct 30 14:11:38.732391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.732376 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a70eaab3-231a-406f-850e-a8184dd7a0d4" containerName="revoke-keys" Oct 30 14:11:38.732391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.732391 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70eaab3-231a-406f-850e-a8184dd7a0d4" containerName="revoke-keys" Oct 30 14:11:38.732545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.732536 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70eaab3-231a-406f-850e-a8184dd7a0d4" containerName="revoke-keys" Oct 30 14:11:38.737256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.737232 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.755755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.755731 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db"] Oct 30 14:11:38.761939 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.761913 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:11:38.762769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.762745 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:11:38.851370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.851340 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqm9\" (UniqueName: \"kubernetes.io/projected/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-kube-api-access-hdqm9\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.851481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.851376 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.851526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.851511 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.953222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.953167 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqm9\" (UniqueName: \"kubernetes.io/projected/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-kube-api-access-hdqm9\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.953369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.953235 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.953369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.953310 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.953998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.953978 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.955808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.955785 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:38.979535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:38.979513 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqm9\" (UniqueName: \"kubernetes.io/projected/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-kube-api-access-hdqm9\") pod \"maas-api-revoke-sub-76275419-765db\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:39.053763 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:39.053705 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:39.358407 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:11:39.358375 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19ca504_ed6b_4c30_a3e6_fb77c38b0433.slice/crio-81e64b99befe6dd12fee70f8f3e1ececeec1c31e37b0d507f2f1c28876cd4433 WatchSource:0}: Error finding container 81e64b99befe6dd12fee70f8f3e1ececeec1c31e37b0d507f2f1c28876cd4433: Status 404 returned error can't find the container with id 81e64b99befe6dd12fee70f8f3e1ececeec1c31e37b0d507f2f1c28876cd4433 Oct 30 14:11:39.360650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:39.360627 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db"] Oct 30 14:11:39.479428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:39.479389 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" event={"ID":"d19ca504-ed6b-4c30-a3e6-fb77c38b0433","Type":"ContainerStarted","Data":"81e64b99befe6dd12fee70f8f3e1ececeec1c31e37b0d507f2f1c28876cd4433"} Oct 30 14:11:40.488324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:40.488234 5824 generic.go:356] "Generic (PLEG): container finished" podID="d19ca504-ed6b-4c30-a3e6-fb77c38b0433" containerID="9f682971ef0f8fdb35b4d2ee1fc2f65dfe254806407009f2d8c0ce73ebe93e30" exitCode=0 Oct 30 14:11:40.488682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:40.488333 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" event={"ID":"d19ca504-ed6b-4c30-a3e6-fb77c38b0433","Type":"ContainerDied","Data":"9f682971ef0f8fdb35b4d2ee1fc2f65dfe254806407009f2d8c0ce73ebe93e30"} Oct 30 14:11:41.666968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.666938 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:41.801870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.801770 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-kube-api-access-hdqm9\" (UniqueName: \"kubernetes.io/projected/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-kube-api-access-hdqm9\") pod \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " Oct 30 14:11:41.802061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.802040 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-openshift-ca\") pod \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " Oct 30 14:11:41.802168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.802149 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-xks-ca\") pod \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\" (UID: \"d19ca504-ed6b-4c30-a3e6-fb77c38b0433\") " Oct 30 14:11:41.802482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.802451 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-openshift-ca" pod "d19ca504-ed6b-4c30-a3e6-fb77c38b0433" (UID: "d19ca504-ed6b-4c30-a3e6-fb77c38b0433"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:41.803008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.802982 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:41.804592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.804560 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-xks-ca" pod "d19ca504-ed6b-4c30-a3e6-fb77c38b0433" (UID: "d19ca504-ed6b-4c30-a3e6-fb77c38b0433"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:41.804754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.804734 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-kube-api-access-hdqm9" pod "d19ca504-ed6b-4c30-a3e6-fb77c38b0433" (UID: "d19ca504-ed6b-4c30-a3e6-fb77c38b0433"). InnerVolumeSpecName "kube-api-access-hdqm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:41.905089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.905057 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdqm9\" (UniqueName: \"kubernetes.io/projected/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-kube-api-access-hdqm9\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:41.905089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:41.905090 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19ca504-ed6b-4c30-a3e6-fb77c38b0433-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:42.504596 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:42.504566 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" Oct 30 14:11:42.504776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:42.504600 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db" event={"ID":"d19ca504-ed6b-4c30-a3e6-fb77c38b0433","Type":"ContainerDied","Data":"81e64b99befe6dd12fee70f8f3e1ececeec1c31e37b0d507f2f1c28876cd4433"} Oct 30 14:11:42.504776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:42.504645 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e64b99befe6dd12fee70f8f3e1ececeec1c31e37b0d507f2f1c28876cd4433" Oct 30 14:11:49.094327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.094296 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv"] Oct 30 14:11:49.094998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.094983 5824 cpu_manager.go:414] "containerMap: removing container" podUID="d19ca504-ed6b-4c30-a3e6-fb77c38b0433" containerName="revoke-keys" Oct 30 14:11:49.094998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.094998 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19ca504-ed6b-4c30-a3e6-fb77c38b0433" containerName="revoke-keys" Oct 30 14:11:49.095148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.095138 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19ca504-ed6b-4c30-a3e6-fb77c38b0433" containerName="revoke-keys" Oct 30 14:11:49.098866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.098848 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.104064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.104038 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.104269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.104253 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccfg\" (UniqueName: \"kubernetes.io/projected/66e79b06-f257-4c49-b57c-315d27cb7548-kube-api-access-hccfg\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.104321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.104292 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.117385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.117355 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:11:49.117493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.117447 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:11:49.123917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.123890 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv"] Oct 30 14:11:49.205491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.205461 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.205685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.205670 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccfg\" (UniqueName: \"kubernetes.io/projected/66e79b06-f257-4c49-b57c-315d27cb7548-kube-api-access-hccfg\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.205759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.205722 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.206122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.206099 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.208411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.208388 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.231963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.231940 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccfg\" (UniqueName: \"kubernetes.io/projected/66e79b06-f257-4c49-b57c-315d27cb7548-kube-api-access-hccfg\") pod \"maas-api-revoke-sub-7232e9b6-dxdfv\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.414495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.414421 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:49.729957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:49.729924 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv"] Oct 30 14:11:50.583713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:50.583673 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" event={"ID":"66e79b06-f257-4c49-b57c-315d27cb7548","Type":"ContainerStarted","Data":"881b34ec0a84e41cc73df3bdcc8bfbd688750fba9f086d5a619bdde782677746"} Oct 30 14:11:50.584072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:50.583724 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" event={"ID":"66e79b06-f257-4c49-b57c-315d27cb7548","Type":"ContainerStarted","Data":"5af8c50bc99bd8df313608a516b58f2e940ecb5a46c703ebc53e979f0661e74e"} Oct 30 14:11:50.632692 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:50.632645 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" podStartSLOduration=1.632632379 podStartE2EDuration="1.632632379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:50.627704015 +0000 UTC m=+1340.428209472" watchObservedRunningTime="2026-10-30 14:11:50.632632379 +0000 UTC m=+1340.433137823" Oct 30 14:11:51.593762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:51.593729 5824 generic.go:356] "Generic (PLEG): container finished" podID="66e79b06-f257-4c49-b57c-315d27cb7548" containerID="881b34ec0a84e41cc73df3bdcc8bfbd688750fba9f086d5a619bdde782677746" exitCode=0 Oct 30 14:11:51.594160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:51.593813 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" event={"ID":"66e79b06-f257-4c49-b57c-315d27cb7548","Type":"ContainerDied","Data":"881b34ec0a84e41cc73df3bdcc8bfbd688750fba9f086d5a619bdde782677746"} Oct 30 14:11:52.802600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.802573 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:52.855939 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.855904 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-xks-ca\") pod \"66e79b06-f257-4c49-b57c-315d27cb7548\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " Oct 30 14:11:52.856204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.856167 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e79b06-f257-4c49-b57c-315d27cb7548-kube-api-access-hccfg\" (UniqueName: \"kubernetes.io/projected/66e79b06-f257-4c49-b57c-315d27cb7548-kube-api-access-hccfg\") pod \"66e79b06-f257-4c49-b57c-315d27cb7548\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " Oct 30 14:11:52.856407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.856386 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-openshift-ca\") pod \"66e79b06-f257-4c49-b57c-315d27cb7548\" (UID: \"66e79b06-f257-4c49-b57c-315d27cb7548\") " Oct 30 14:11:52.856805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.856778 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-openshift-ca" pod "66e79b06-f257-4c49-b57c-315d27cb7548" (UID: "66e79b06-f257-4c49-b57c-315d27cb7548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:11:52.857283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.857259 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.859535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.859511 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e79b06-f257-4c49-b57c-315d27cb7548-kube-api-access-hccfg" pod "66e79b06-f257-4c49-b57c-315d27cb7548" (UID: "66e79b06-f257-4c49-b57c-315d27cb7548"). InnerVolumeSpecName "kube-api-access-hccfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:11:52.861340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.861307 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-xks-ca" pod "66e79b06-f257-4c49-b57c-315d27cb7548" (UID: "66e79b06-f257-4c49-b57c-315d27cb7548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:11:52.958811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.958778 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e79b06-f257-4c49-b57c-315d27cb7548-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:52.958811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:52.958806 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hccfg\" (UniqueName: \"kubernetes.io/projected/66e79b06-f257-4c49-b57c-315d27cb7548-kube-api-access-hccfg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:11:53.613906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:53.613857 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" event={"ID":"66e79b06-f257-4c49-b57c-315d27cb7548","Type":"ContainerDied","Data":"5af8c50bc99bd8df313608a516b58f2e940ecb5a46c703ebc53e979f0661e74e"} Oct 30 14:11:53.614091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:53.613985 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af8c50bc99bd8df313608a516b58f2e940ecb5a46c703ebc53e979f0661e74e" Oct 30 14:11:53.614171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:53.614133 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv" Oct 30 14:11:57.135365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.135318 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 14:11:57.147298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.147260 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:11:57.203066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.203012 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:11:57.564214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.554614 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.580570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.580513 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.717944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.717902 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:57.736397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.736358 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:11:57.736918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.736890 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:11:57.864352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.864258 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:11:57.893220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:57.888255 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.084230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.075392 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.150266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.148395 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.172259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.172231 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:11:58.188999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.188950 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.192723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.189621 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:11:58.211563 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.203767 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.212135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.205021 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.212400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.212385 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:11:58.580903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.554476 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.591594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.591273 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.725582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.724539 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Oct 30 14:11:58.749719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.749664 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.894267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.889552 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.896988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.896949 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.905306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.905243 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:11:58.910397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.910361 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:11:59.186417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:58.926476 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:11:59.192716 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.192689 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:11:59.192931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.192917 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:11:59.193089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.193076 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:11:59.193234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.193221 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:11:59.203151 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.201549 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 14:11:59.215468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.215434 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:11:59.246650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.246621 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:11:59.286810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.286773 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:11:59.374691 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.374661 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:11:59.393570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.388184 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:11:59.393570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.389492 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:11:59.403658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.403630 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:11:59.430865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:11:59.430840 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:12:01.151314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.151284 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7"] Oct 30 14:12:01.152042 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.152025 5824 cpu_manager.go:414] "containerMap: removing container" podUID="66e79b06-f257-4c49-b57c-315d27cb7548" containerName="revoke-keys" Oct 30 14:12:01.152089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.152044 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e79b06-f257-4c49-b57c-315d27cb7548" containerName="revoke-keys" Oct 30 14:12:01.152153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.152144 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e79b06-f257-4c49-b57c-315d27cb7548" containerName="revoke-keys" Oct 30 14:12:01.157177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.157156 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.174165 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.174137 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7"] Oct 30 14:12:01.181547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.181523 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:12:01.181684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.181589 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:12:01.202133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.202097 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgk5m\" (UniqueName: \"kubernetes.io/projected/159bbda6-cd39-4582-9217-b3b2c7c11a93-kube-api-access-dgk5m\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.202464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.202442 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.202604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.202574 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.303797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.303765 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.303970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.303842 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.304015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.303970 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgk5m\" (UniqueName: \"kubernetes.io/projected/159bbda6-cd39-4582-9217-b3b2c7c11a93-kube-api-access-dgk5m\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.304556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.304534 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.306175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.306149 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.330480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.330455 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgk5m\" (UniqueName: \"kubernetes.io/projected/159bbda6-cd39-4582-9217-b3b2c7c11a93-kube-api-access-dgk5m\") pod \"maas-api-revoke-sub-fd80aa4b-cqxn7\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.471968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.471943 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:01.784979 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:12:01.784949 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159bbda6_cd39_4582_9217_b3b2c7c11a93.slice/crio-4bc56fa33361084e644b286d69b27a995e03f4591ae816a2eaba28a0faa69c47 WatchSource:0}: Error finding container 4bc56fa33361084e644b286d69b27a995e03f4591ae816a2eaba28a0faa69c47: Status 404 returned error can't find the container with id 4bc56fa33361084e644b286d69b27a995e03f4591ae816a2eaba28a0faa69c47 Oct 30 14:12:01.788968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.788940 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7"] Oct 30 14:12:01.957145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:01.957111 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" event={"ID":"159bbda6-cd39-4582-9217-b3b2c7c11a93","Type":"ContainerStarted","Data":"4bc56fa33361084e644b286d69b27a995e03f4591ae816a2eaba28a0faa69c47"} Oct 30 14:12:02.968182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:02.968135 5824 generic.go:356] "Generic (PLEG): container finished" podID="159bbda6-cd39-4582-9217-b3b2c7c11a93" containerID="d031d17858449854e8ff5d7f516df2b37c5569084f818fc1ef3162884cfb3528" exitCode=0 Oct 30 14:12:02.968685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:02.968260 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" event={"ID":"159bbda6-cd39-4582-9217-b3b2c7c11a93","Type":"ContainerDied","Data":"d031d17858449854e8ff5d7f516df2b37c5569084f818fc1ef3162884cfb3528"} Oct 30 14:12:03.641278 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:12:03.641243 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:12:04.139001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.138979 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:04.248454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.248371 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159bbda6-cd39-4582-9217-b3b2c7c11a93-kube-api-access-dgk5m\" (UniqueName: \"kubernetes.io/projected/159bbda6-cd39-4582-9217-b3b2c7c11a93-kube-api-access-dgk5m\") pod \"159bbda6-cd39-4582-9217-b3b2c7c11a93\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " Oct 30 14:12:04.248617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.248502 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-xks-ca\") pod \"159bbda6-cd39-4582-9217-b3b2c7c11a93\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " Oct 30 14:12:04.248617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.248609 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-openshift-ca\") pod \"159bbda6-cd39-4582-9217-b3b2c7c11a93\" (UID: \"159bbda6-cd39-4582-9217-b3b2c7c11a93\") " Oct 30 14:12:04.249992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.249955 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-openshift-ca" pod "159bbda6-cd39-4582-9217-b3b2c7c11a93" (UID: "159bbda6-cd39-4582-9217-b3b2c7c11a93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:04.250872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.250844 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159bbda6-cd39-4582-9217-b3b2c7c11a93-kube-api-access-dgk5m" pod "159bbda6-cd39-4582-9217-b3b2c7c11a93" (UID: "159bbda6-cd39-4582-9217-b3b2c7c11a93"). InnerVolumeSpecName "kube-api-access-dgk5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:04.251800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.251775 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-xks-ca" pod "159bbda6-cd39-4582-9217-b3b2c7c11a93" (UID: "159bbda6-cd39-4582-9217-b3b2c7c11a93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:04.351709 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.351672 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgk5m\" (UniqueName: \"kubernetes.io/projected/159bbda6-cd39-4582-9217-b3b2c7c11a93-kube-api-access-dgk5m\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:04.351709 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.351708 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:04.351927 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.351724 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/159bbda6-cd39-4582-9217-b3b2c7c11a93-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:04.849310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.849275 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx"] Oct 30 14:12:04.850003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.849988 5824 cpu_manager.go:414] "containerMap: removing container" podUID="159bbda6-cd39-4582-9217-b3b2c7c11a93" containerName="revoke-keys" Oct 30 14:12:04.850058 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.850004 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159bbda6-cd39-4582-9217-b3b2c7c11a93" containerName="revoke-keys" Oct 30 14:12:04.850150 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.850140 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="159bbda6-cd39-4582-9217-b3b2c7c11a93" containerName="revoke-keys" Oct 30 14:12:04.854093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.854075 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:04.873478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.873447 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx"] Oct 30 14:12:04.961249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.961214 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkr5n\" (UniqueName: \"kubernetes.io/projected/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-kube-api-access-pkr5n\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:04.961428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.961277 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:04.961508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.961494 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:04.986771 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.986739 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" event={"ID":"159bbda6-cd39-4582-9217-b3b2c7c11a93","Type":"ContainerDied","Data":"4bc56fa33361084e644b286d69b27a995e03f4591ae816a2eaba28a0faa69c47"} Oct 30 14:12:04.986909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.986791 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bc56fa33361084e644b286d69b27a995e03f4591ae816a2eaba28a0faa69c47" Oct 30 14:12:04.986909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:04.986745 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7" Oct 30 14:12:05.063204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.063166 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkr5n\" (UniqueName: \"kubernetes.io/projected/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-kube-api-access-pkr5n\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:05.063327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.063219 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:05.063371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.063357 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:05.063892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.063870 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:05.065694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.065668 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:05.089276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.089252 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkr5n\" (UniqueName: \"kubernetes.io/projected/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-kube-api-access-pkr5n\") pod \"maas-api-revoke-sub-236e8450-z2spx\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:05.168820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.168750 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:05.476374 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:12:05.476342 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e9fad3_48c8_45f3_9346_4ff5bab94be9.slice/crio-86caeaa1b6f958344a4b7f0d77135447d4065e55a3fbb1d508dd69f7b6028db0 WatchSource:0}: Error finding container 86caeaa1b6f958344a4b7f0d77135447d4065e55a3fbb1d508dd69f7b6028db0: Status 404 returned error can't find the container with id 86caeaa1b6f958344a4b7f0d77135447d4065e55a3fbb1d508dd69f7b6028db0 Oct 30 14:12:05.477799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.477772 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx"] Oct 30 14:12:05.997136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.997104 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" event={"ID":"f4e9fad3-48c8-45f3-9346-4ff5bab94be9","Type":"ContainerStarted","Data":"c65a9c6de34a80051b48c9798ebc6c83cca048996f5f03dc32e97d72c7972e1e"} Oct 30 14:12:05.997297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:05.997157 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" event={"ID":"f4e9fad3-48c8-45f3-9346-4ff5bab94be9","Type":"ContainerStarted","Data":"86caeaa1b6f958344a4b7f0d77135447d4065e55a3fbb1d508dd69f7b6028db0"} Oct 30 14:12:06.069779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:06.069698 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" podStartSLOduration=2.069684259 podStartE2EDuration="2.069684259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:06.066010319 +0000 UTC m=+1355.866515754" watchObservedRunningTime="2026-10-30 14:12:06.069684259 +0000 UTC m=+1355.870189703" Oct 30 14:12:07.005708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:07.005671 5824 generic.go:356] "Generic (PLEG): container finished" podID="f4e9fad3-48c8-45f3-9346-4ff5bab94be9" containerID="c65a9c6de34a80051b48c9798ebc6c83cca048996f5f03dc32e97d72c7972e1e" exitCode=0 Oct 30 14:12:07.006090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:07.005718 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" event={"ID":"f4e9fad3-48c8-45f3-9346-4ff5bab94be9","Type":"ContainerDied","Data":"c65a9c6de34a80051b48c9798ebc6c83cca048996f5f03dc32e97d72c7972e1e"} Oct 30 14:12:08.180850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.180820 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:08.306844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.306772 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-xks-ca\") pod \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " Oct 30 14:12:08.306844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.306819 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-kube-api-access-pkr5n\" (UniqueName: \"kubernetes.io/projected/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-kube-api-access-pkr5n\") pod \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " Oct 30 14:12:08.307041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.306935 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-openshift-ca\") pod \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\" (UID: \"f4e9fad3-48c8-45f3-9346-4ff5bab94be9\") " Oct 30 14:12:08.307299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.307277 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-openshift-ca" pod "f4e9fad3-48c8-45f3-9346-4ff5bab94be9" (UID: "f4e9fad3-48c8-45f3-9346-4ff5bab94be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:08.307755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.307738 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:08.308909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.308889 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-kube-api-access-pkr5n" pod "f4e9fad3-48c8-45f3-9346-4ff5bab94be9" (UID: "f4e9fad3-48c8-45f3-9346-4ff5bab94be9"). InnerVolumeSpecName "kube-api-access-pkr5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:08.308956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.308893 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-xks-ca" pod "f4e9fad3-48c8-45f3-9346-4ff5bab94be9" (UID: "f4e9fad3-48c8-45f3-9346-4ff5bab94be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:08.409837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.409803 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:08.409837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:08.409829 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkr5n\" (UniqueName: \"kubernetes.io/projected/f4e9fad3-48c8-45f3-9346-4ff5bab94be9-kube-api-access-pkr5n\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:09.023046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:09.023009 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" Oct 30 14:12:09.023046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:09.023013 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx" event={"ID":"f4e9fad3-48c8-45f3-9346-4ff5bab94be9","Type":"ContainerDied","Data":"86caeaa1b6f958344a4b7f0d77135447d4065e55a3fbb1d508dd69f7b6028db0"} Oct 30 14:12:09.023286 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:09.023072 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86caeaa1b6f958344a4b7f0d77135447d4065e55a3fbb1d508dd69f7b6028db0" Oct 30 14:12:09.043206 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:09.043172 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6"] Oct 30 14:12:09.056991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:09.056958 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp"] Oct 30 14:12:09.060881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:09.060858 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d18926a-4tsw6"] Oct 30 14:12:09.064381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:09.064361 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f19bc01-vzpqp"] Oct 30 14:12:10.325518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:10.322485 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c182b11-31fe-410a-b3f5-9130ddac9552" path="/var/lib/kubelet/pods/3c182b11-31fe-410a-b3f5-9130ddac9552/volumes" Oct 30 14:12:10.325518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:10.323341 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a" path="/var/lib/kubelet/pods/f0a4ac4f-fc4f-4bb5-86ad-2a644b3ce56a/volumes" Oct 30 14:12:17.044924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:17.044841 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn"] Oct 30 14:12:17.060516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:17.060484 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe6e47c-l66dn"] Oct 30 14:12:18.318764 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:18.318730 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e376886-9e11-4ed3-a546-1249ec38b620" path="/var/lib/kubelet/pods/0e376886-9e11-4ed3-a546-1249ec38b620/volumes" Oct 30 14:12:24.127211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:24.126857 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 14:12:33.642477 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:12:33.642437 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:12:33.755018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.754984 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66"] Oct 30 14:12:33.756176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.756152 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f4e9fad3-48c8-45f3-9346-4ff5bab94be9" containerName="revoke-keys" Oct 30 14:12:33.756363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.756348 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e9fad3-48c8-45f3-9346-4ff5bab94be9" containerName="revoke-keys" Oct 30 14:12:33.756734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.756706 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e9fad3-48c8-45f3-9346-4ff5bab94be9" containerName="revoke-keys" Oct 30 14:12:33.763248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.763224 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:33.775397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.775368 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66"] Oct 30 14:12:33.782988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.782965 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:12:33.783226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.783168 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:12:33.872857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.872818 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:33.873061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.872984 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:33.873173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.873157 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxfg\" (UniqueName: \"kubernetes.io/projected/b7518f76-e056-4ce2-8462-f693ad20cedf-kube-api-access-vsxfg\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:33.976995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.975416 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:33.976995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.975606 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxfg\" (UniqueName: \"kubernetes.io/projected/b7518f76-e056-4ce2-8462-f693ad20cedf-kube-api-access-vsxfg\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:33.976995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.975767 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:33.976995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.976545 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:33.982960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:33.982935 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:34.002408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.002384 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxfg\" (UniqueName: \"kubernetes.io/projected/b7518f76-e056-4ce2-8462-f693ad20cedf-kube-api-access-vsxfg\") pod \"maas-api-revoke-sub-48f2bd82-n7w66\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:34.023638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.023613 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb"] Oct 30 14:12:34.028304 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.028280 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:34.062402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.048665 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb"] Oct 30 14:12:34.062402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.049123 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-25917498\"" Oct 30 14:12:34.062402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.049330 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-25917498-dockercfg-lw49k\"" Oct 30 14:12:34.077841 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.077801 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdnm4\" (UniqueName: \"kubernetes.io/projected/4c99baa2-49ef-4912-bf3d-5f8c613398a5-kube-api-access-vdnm4\") pod \"payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb\" (UID: \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:34.078235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.078209 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c99baa2-49ef-4912-bf3d-5f8c613398a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb\" (UID: \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:34.092085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.092051 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:34.189212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.185146 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdnm4\" (UniqueName: \"kubernetes.io/projected/4c99baa2-49ef-4912-bf3d-5f8c613398a5-kube-api-access-vdnm4\") pod \"payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb\" (UID: \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:34.189212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.185380 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c99baa2-49ef-4912-bf3d-5f8c613398a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb\" (UID: \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:34.189212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.186576 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c99baa2-49ef-4912-bf3d-5f8c613398a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb\" (UID: \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:34.236509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.229727 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdnm4\" (UniqueName: \"kubernetes.io/projected/4c99baa2-49ef-4912-bf3d-5f8c613398a5-kube-api-access-vdnm4\") pod \"payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb\" (UID: \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:34.313344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.303299 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5"] Oct 30 14:12:34.325892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.325138 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:34.344663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.343513 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:34.344663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.344314 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5"] Oct 30 14:12:34.403237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.403151 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41043006-3e00-4724-bef9-415bb5f95c57-plugins-config-volume\") pod \"payload-processing-e2e-ait-25917498-5b656767dd-2xdq5\" (UID: \"41043006-3e00-4724-bef9-415bb5f95c57\") " pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:34.403237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.403241 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngqm\" (UniqueName: \"kubernetes.io/projected/41043006-3e00-4724-bef9-415bb5f95c57-kube-api-access-xngqm\") pod \"payload-processing-e2e-ait-25917498-5b656767dd-2xdq5\" (UID: \"41043006-3e00-4724-bef9-415bb5f95c57\") " pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:34.505596 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.505563 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41043006-3e00-4724-bef9-415bb5f95c57-plugins-config-volume\") pod \"payload-processing-e2e-ait-25917498-5b656767dd-2xdq5\" (UID: \"41043006-3e00-4724-bef9-415bb5f95c57\") " pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:34.505733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.505633 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngqm\" (UniqueName: \"kubernetes.io/projected/41043006-3e00-4724-bef9-415bb5f95c57-kube-api-access-xngqm\") pod \"payload-processing-e2e-ait-25917498-5b656767dd-2xdq5\" (UID: \"41043006-3e00-4724-bef9-415bb5f95c57\") " pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:34.507204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.507162 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41043006-3e00-4724-bef9-415bb5f95c57-plugins-config-volume\") pod \"payload-processing-e2e-ait-25917498-5b656767dd-2xdq5\" (UID: \"41043006-3e00-4724-bef9-415bb5f95c57\") " pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:34.512805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.512782 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66"] Oct 30 14:12:34.536753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.536722 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngqm\" (UniqueName: \"kubernetes.io/projected/41043006-3e00-4724-bef9-415bb5f95c57-kube-api-access-xngqm\") pod \"payload-processing-e2e-ait-25917498-5b656767dd-2xdq5\" (UID: \"41043006-3e00-4724-bef9-415bb5f95c57\") " pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:34.643727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.643681 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:34.674273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:34.674248 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb"] Oct 30 14:12:34.745098 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:12:34.745062 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c99baa2_49ef_4912_bf3d_5f8c613398a5.slice/crio-4753030f5b9f637b92ae52eda488b739c3d0f1043a20ce3725b0507494a0923f WatchSource:0}: Error finding container 4753030f5b9f637b92ae52eda488b739c3d0f1043a20ce3725b0507494a0923f: Status 404 returned error can't find the container with id 4753030f5b9f637b92ae52eda488b739c3d0f1043a20ce3725b0507494a0923f Oct 30 14:12:35.068601 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:12:35.068574 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41043006_3e00_4724_bef9_415bb5f95c57.slice/crio-e99e590ffd92421b86488afbed3d135cface4f4d5a63b308cebb7c654a7c563c WatchSource:0}: Error finding container e99e590ffd92421b86488afbed3d135cface4f4d5a63b308cebb7c654a7c563c: Status 404 returned error can't find the container with id e99e590ffd92421b86488afbed3d135cface4f4d5a63b308cebb7c654a7c563c Oct 30 14:12:35.070461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.070439 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5"] Oct 30 14:12:35.368316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.368220 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" event={"ID":"4c99baa2-49ef-4912-bf3d-5f8c613398a5","Type":"ContainerStarted","Data":"1f3c691db510263a5fafd22fcdbda41880b5457ef1f985df730141ed3f75dff8"} Oct 30 14:12:35.368316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.368286 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" event={"ID":"4c99baa2-49ef-4912-bf3d-5f8c613398a5","Type":"ContainerStarted","Data":"4753030f5b9f637b92ae52eda488b739c3d0f1043a20ce3725b0507494a0923f"} Oct 30 14:12:35.368544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.368325 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:35.370267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.370225 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" event={"ID":"b7518f76-e056-4ce2-8462-f693ad20cedf","Type":"ContainerStarted","Data":"c1ade2f123003403ac215a8bbb4c53e76922c12d8f714c6c78f694ce9a890ba0"} Oct 30 14:12:35.370496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.370276 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" event={"ID":"b7518f76-e056-4ce2-8462-f693ad20cedf","Type":"ContainerStarted","Data":"ab3a6e917aeee355e1fdc794b1c9e5b8fa46dba399e559f295ef2eab56674484"} Oct 30 14:12:35.372318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.372231 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" event={"ID":"41043006-3e00-4724-bef9-415bb5f95c57","Type":"ContainerStarted","Data":"33c352ba08ac1217fd42db9155ff943726151e19272eb03c488230fe1879822d"} Oct 30 14:12:35.372318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.372290 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" event={"ID":"41043006-3e00-4724-bef9-415bb5f95c57","Type":"ContainerStarted","Data":"e99e590ffd92421b86488afbed3d135cface4f4d5a63b308cebb7c654a7c563c"} Oct 30 14:12:35.372468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.372388 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:35.420730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.420669 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" podStartSLOduration=2.420654868 podStartE2EDuration="2.420654868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:35.413871021 +0000 UTC m=+1385.214376464" watchObservedRunningTime="2026-10-30 14:12:35.420654868 +0000 UTC m=+1385.221160306" Oct 30 14:12:35.468869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.468804 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" podStartSLOduration=1.4687869390000001 podStartE2EDuration="1.468786939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:35.461611963 +0000 UTC m=+1385.262117408" watchObservedRunningTime="2026-10-30 14:12:35.468786939 +0000 UTC m=+1385.269292383" Oct 30 14:12:35.510064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.510007 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" podStartSLOduration=2.509992702 podStartE2EDuration="2.509992702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:35.505733598 +0000 UTC m=+1385.306239042" watchObservedRunningTime="2026-10-30 14:12:35.509992702 +0000 UTC m=+1385.310498147" Oct 30 14:12:35.707906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.707873 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9"] Oct 30 14:12:35.712553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.712531 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.731663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.731467 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-da21809f-serving-cert\"" Oct 30 14:12:35.739605 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.739575 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9"] Oct 30 14:12:35.835658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.835623 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-metrics-certs\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.835828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.835790 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.835922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.835902 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9sf5\" (UniqueName: \"kubernetes.io/projected/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-kube-api-access-w9sf5\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.835978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.835931 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-maas-api-tls\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.937555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.937522 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9sf5\" (UniqueName: \"kubernetes.io/projected/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-kube-api-access-w9sf5\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.937555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.937556 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-maas-api-tls\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.937779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.937711 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-metrics-certs\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.937836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.937820 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.938419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.938395 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.940210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.940171 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-maas-api-tls\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.940305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.940271 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-metrics-certs\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:35.963618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:35.963591 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9sf5\" (UniqueName: \"kubernetes.io/projected/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-kube-api-access-w9sf5\") pod \"maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:36.032573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:36.032535 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:36.350696 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:12:36.350667 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode613965e_4ed2_4cb6_812f_e2d6596aa4a3.slice/crio-81a96d013b547a38443e263785d88ae669d5b0362b6602ccc77b42779cbdf7be WatchSource:0}: Error finding container 81a96d013b547a38443e263785d88ae669d5b0362b6602ccc77b42779cbdf7be: Status 404 returned error can't find the container with id 81a96d013b547a38443e263785d88ae669d5b0362b6602ccc77b42779cbdf7be Oct 30 14:12:36.353002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:36.352980 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9"] Oct 30 14:12:36.382915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:36.382883 5824 generic.go:356] "Generic (PLEG): container finished" podID="b7518f76-e056-4ce2-8462-f693ad20cedf" containerID="c1ade2f123003403ac215a8bbb4c53e76922c12d8f714c6c78f694ce9a890ba0" exitCode=0 Oct 30 14:12:36.383061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:36.383037 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" event={"ID":"b7518f76-e056-4ce2-8462-f693ad20cedf","Type":"ContainerDied","Data":"c1ade2f123003403ac215a8bbb4c53e76922c12d8f714c6c78f694ce9a890ba0"} Oct 30 14:12:36.384631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:36.384608 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" event={"ID":"e613965e-4ed2-4cb6-812f-e2d6596aa4a3","Type":"ContainerStarted","Data":"81a96d013b547a38443e263785d88ae669d5b0362b6602ccc77b42779cbdf7be"} Oct 30 14:12:37.395603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.395565 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" event={"ID":"e613965e-4ed2-4cb6-812f-e2d6596aa4a3","Type":"ContainerStarted","Data":"f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653"} Oct 30 14:12:37.445058 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.445002 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" podStartSLOduration=2.057262187 podStartE2EDuration="2.444988896s" totalImagesPullingTime="387.726709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:37.442378134 +0000 UTC m=+1387.242883582" watchObservedRunningTime="2026-10-30 14:12:37.444988896 +0000 UTC m=+1387.245494339" Oct 30 14:12:37.599576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.599549 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:37.780357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.780322 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-openshift-ca\") pod \"b7518f76-e056-4ce2-8462-f693ad20cedf\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " Oct 30 14:12:37.780560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.780387 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-xks-ca\") pod \"b7518f76-e056-4ce2-8462-f693ad20cedf\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " Oct 30 14:12:37.780743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.780710 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-openshift-ca" pod "b7518f76-e056-4ce2-8462-f693ad20cedf" (UID: "b7518f76-e056-4ce2-8462-f693ad20cedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:37.780743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.780732 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7518f76-e056-4ce2-8462-f693ad20cedf-kube-api-access-vsxfg\" (UniqueName: \"kubernetes.io/projected/b7518f76-e056-4ce2-8462-f693ad20cedf-kube-api-access-vsxfg\") pod \"b7518f76-e056-4ce2-8462-f693ad20cedf\" (UID: \"b7518f76-e056-4ce2-8462-f693ad20cedf\") " Oct 30 14:12:37.781423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.781397 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:37.782849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.782820 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-xks-ca" pod "b7518f76-e056-4ce2-8462-f693ad20cedf" (UID: "b7518f76-e056-4ce2-8462-f693ad20cedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:12:37.782956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.782905 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7518f76-e056-4ce2-8462-f693ad20cedf-kube-api-access-vsxfg" pod "b7518f76-e056-4ce2-8462-f693ad20cedf" (UID: "b7518f76-e056-4ce2-8462-f693ad20cedf"). InnerVolumeSpecName "kube-api-access-vsxfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:37.883207 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.883154 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsxfg\" (UniqueName: \"kubernetes.io/projected/b7518f76-e056-4ce2-8462-f693ad20cedf-kube-api-access-vsxfg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:37.883207 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:37.883205 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7518f76-e056-4ce2-8462-f693ad20cedf-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:38.406003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:38.405972 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" Oct 30 14:12:38.406489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:38.406007 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66" event={"ID":"b7518f76-e056-4ce2-8462-f693ad20cedf","Type":"ContainerDied","Data":"ab3a6e917aeee355e1fdc794b1c9e5b8fa46dba399e559f295ef2eab56674484"} Oct 30 14:12:38.406489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:38.406056 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab3a6e917aeee355e1fdc794b1c9e5b8fa46dba399e559f295ef2eab56674484" Oct 30 14:12:38.406489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:38.406312 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:44.422373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:44.422347 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:12:46.388467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:46.388438 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:12:46.388868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:46.388506 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:12:49.389135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:49.388718 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx"] Oct 30 14:12:49.389135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:49.388975 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" containerID="cri-o://ba27233119cd9df1ea668ba28cd50f080614765b1607ff91336c6f053721de19" gracePeriod=30 Oct 30 14:12:52.079213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.078310 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.152222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.148273 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 14:12:52.235213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.234344 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.257239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.256285 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.288213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.287476 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.390264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.389332 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.413214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.390569 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.413214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.391287 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.413214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.410465 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.580234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.563292 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.584220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.582277 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:52.741492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:52.740292 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.083405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.083299 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.084543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.084519 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:12:53.179213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.174280 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.209214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.182739 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:12:53.268842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.237992 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:12:53.326289 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.269102 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.388746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.295098 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:12:53.388746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.332330 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:12:53.389960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.389936 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:12:53.425598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.420343 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:12:53.442846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.442801 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.460173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.443501 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.460173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.455403 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:12:53.460173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.443534 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:12:53.520211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.466215 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:12:53.520211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.443563 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 14:12:53.520211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.467174 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:12:53.520211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.492361 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.520211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.496843 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:12:53.584368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.575649 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.595782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.585640 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:12:53.595782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.585781 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 14:12:53.618232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.601700 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:12:53.798216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.748595 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:53.798216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:53.749252 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:12:54.130264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.127808 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:12:54.155218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.135986 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 14:12:54.155218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.136840 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.257215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.177274 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.286213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.273704 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:12:54.304214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.300283 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:12:54.383822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.361299 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:12:54.477000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.470924 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:12:54.477000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.474412 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.477000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.474449 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.530940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.530906 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 14:12:54.546935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.541297 5824 patch_prober.go:29] interesting pod/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:12:54.546935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.541341 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.582373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.582336 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.609148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.593167 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:12:54.610064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.610043 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:12:54.727806 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.727762 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:12:54.872126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.870259 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" podUID="7c410a7f-e3b9-485d-bbd4-2b3d83c2d297" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 14:12:54.973296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.935287 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:12:54.973296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.935374 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:12:54.973296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.935455 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:12:54.973296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.935510 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:12:54.973296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.935562 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:12:54.973296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.939431 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:12:54.973296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.958748 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz" Oct 30 14:12:54.973296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:54.968031 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:12:55.098294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.073555 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j"] Oct 30 14:12:55.125149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.124312 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:12:55.126351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.126326 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c0aec2f-mcc7j"] Oct 30 14:12:55.320209 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.319769 5824 generic.go:356] "Generic (PLEG): container finished" podID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerID="ba27233119cd9df1ea668ba28cd50f080614765b1607ff91336c6f053721de19" exitCode=0 Oct 30 14:12:55.321211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.320833 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" event={"ID":"7c60bc55-ac39-410b-be5f-247da3bca3fb","Type":"ContainerDied","Data":"ba27233119cd9df1ea668ba28cd50f080614765b1607ff91336c6f053721de19"} Oct 30 14:12:55.536164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.536129 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc"] Oct 30 14:12:55.536410 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.536385 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" containerID="cri-o://b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4" gracePeriod=30 Oct 30 14:12:55.589434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.589406 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:12:55.702310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702275 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-data\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.702508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702340 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-socket\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.702508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702423 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-token\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-token\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.702508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702486 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c60bc55-ac39-410b-be5f-247da3bca3fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c60bc55-ac39-410b-be5f-247da3bca3fb-istiod-ca-cert\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.702710 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702622 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-data" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:55.702710 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702680 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-podinfo\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.702891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702747 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-kube-api-access-bf4m4\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-kube-api-access-bf4m4\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.702891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702785 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-credential-socket\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.703008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.702924 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-certs\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.703208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.703133 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-credential-socket" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:55.703341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.703225 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-certs" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:55.703341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.703288 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c60bc55-ac39-410b-be5f-247da3bca3fb-istiod-ca-cert" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:12:55.703477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.703360 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-envoy\") pod \"7c60bc55-ac39-410b-be5f-247da3bca3fb\" (UID: \"7c60bc55-ac39-410b-be5f-247da3bca3fb\") " Oct 30 14:12:55.704383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.704359 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-socket" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:55.706942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.706915 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:55.707120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.706946 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:55.707120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.706962 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c60bc55-ac39-410b-be5f-247da3bca3fb-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:55.707120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.706978 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:55.707120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.706994 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:55.710471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.710443 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-podinfo" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:12:55.710617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.710573 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-envoy" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:12:55.712233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.712091 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-token" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:55.712571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.712483 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-kube-api-access-bf4m4" pod "7c60bc55-ac39-410b-be5f-247da3bca3fb" (UID: "7c60bc55-ac39-410b-be5f-247da3bca3fb"). InnerVolumeSpecName "kube-api-access-bf4m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:12:55.808955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.808876 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:55.808955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.808909 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:55.808955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.808920 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf4m4\" (UniqueName: \"kubernetes.io/projected/7c60bc55-ac39-410b-be5f-247da3bca3fb-kube-api-access-bf4m4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:55.808955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:55.808930 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c60bc55-ac39-410b-be5f-247da3bca3fb-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:12:56.339237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:56.328465 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01af172e-407c-4f7c-9bdd-e892982339ec" path="/var/lib/kubelet/pods/01af172e-407c-4f7c-9bdd-e892982339ec/volumes" Oct 30 14:12:56.339237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:56.334602 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" event={"ID":"7c60bc55-ac39-410b-be5f-247da3bca3fb","Type":"ContainerDied","Data":"39d1025a7cd12a58e04d6fc20de7a4fdd9330b76ed0737bb92865d97faa59704"} Oct 30 14:12:56.339237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:56.334632 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx" Oct 30 14:12:56.339237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:56.334649 5824 scope.go:122] "RemoveContainer" containerID="ba27233119cd9df1ea668ba28cd50f080614765b1607ff91336c6f053721de19" Oct 30 14:12:56.404211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:56.403529 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx"] Oct 30 14:12:56.412798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:56.412772 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t5tsx"] Oct 30 14:12:58.329919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:12:58.329878 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" path="/var/lib/kubelet/pods/7c60bc55-ac39-410b-be5f-247da3bca3fb/volumes" Oct 30 14:13:00.865901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.865872 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:13:00.890406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.890378 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-data\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.890597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.890579 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-envoy\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.890654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.890636 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-data" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:00.890719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.890685 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-kube-api-access-c5wpt\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-kube-api-access-c5wpt\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.890719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.890715 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-podinfo\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.890897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.890868 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-socket\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.890995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.890944 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istiod-ca-cert\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.890988 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-certs\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.891655 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-credential-socket\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.891692 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-certs" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.891711 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-token\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-token\") pod \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\" (UID: \"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0\") " Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.891875 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-credential-socket" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.893792 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.893826 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.893842 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.891092 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-socket" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.894176 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istiod-ca-cert" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:00.896299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.894384 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-token" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:00.913088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.913064 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-envoy" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:13:00.913824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.913797 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-podinfo" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:13:00.915544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.915517 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-kube-api-access-c5wpt" pod "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" (UID: "1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0"). InnerVolumeSpecName "kube-api-access-c5wpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:00.995514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.995474 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:00.995514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.995514 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:00.995719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.995531 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5wpt\" (UniqueName: \"kubernetes.io/projected/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-kube-api-access-c5wpt\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:00.995719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.995547 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:00.995719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.995562 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:00.995719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:00.995575 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:01.390364 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.390328 5824 generic.go:356] "Generic (PLEG): container finished" podID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerID="b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4" exitCode=0 Oct 30 14:13:01.390651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.390630 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" event={"ID":"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0","Type":"ContainerDied","Data":"b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4"} Oct 30 14:13:01.390723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.390688 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" event={"ID":"1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0","Type":"ContainerDied","Data":"b092f52f0b4b3271cc203d1957961f453bddee1c787315f74a830021203d27df"} Oct 30 14:13:01.390723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.390717 5824 scope.go:122] "RemoveContainer" containerID="b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4" Oct 30 14:13:01.390898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.390884 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc" Oct 30 14:13:01.409887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.409855 5824 scope.go:122] "RemoveContainer" containerID="44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a" Oct 30 14:13:01.422983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.422958 5824 scope.go:122] "RemoveContainer" containerID="b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4" Oct 30 14:13:01.423387 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:13:01.423357 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4\": container with ID starting with b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4 not found: ID does not exist" containerID="b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4" Oct 30 14:13:01.423499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.423396 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4"} err="failed to get container status \"b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4\": rpc error: code = NotFound desc = could not find container \"b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4\": container with ID starting with b522937c186f12283168998d7d024de2286deda9712085e7e016125efc6572d4 not found: ID does not exist" Oct 30 14:13:01.423499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.423418 5824 scope.go:122] "RemoveContainer" containerID="44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a" Oct 30 14:13:01.423860 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:13:01.423830 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a\": container with ID starting with 44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a not found: ID does not exist" containerID="44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a" Oct 30 14:13:01.423922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.423873 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a"} err="failed to get container status \"44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a\": rpc error: code = NotFound desc = could not find container \"44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a\": container with ID starting with 44c36873ff56ec636c753dc29dcc188393302e32c51cadbb0f040591cf89f40a not found: ID does not exist" Oct 30 14:13:01.461331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.461302 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc"] Oct 30 14:13:01.468048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:01.468018 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-f2dhc"] Oct 30 14:13:02.319105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:02.319076 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" path="/var/lib/kubelet/pods/1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0/volumes" Oct 30 14:13:03.641937 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:13:03.641900 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:13:19.541026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.540982 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq"] Oct 30 14:13:19.542172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542142 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" Oct 30 14:13:19.542172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542172 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" Oct 30 14:13:19.542500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542211 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" Oct 30 14:13:19.542500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542220 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" Oct 30 14:13:19.542500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542257 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" Oct 30 14:13:19.542500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542265 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" Oct 30 14:13:19.542500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542290 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b7518f76-e056-4ce2-8462-f693ad20cedf" containerName="revoke-keys" Oct 30 14:13:19.542500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542298 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7518f76-e056-4ce2-8462-f693ad20cedf" containerName="revoke-keys" Oct 30 14:13:19.542500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542496 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c60bc55-ac39-410b-be5f-247da3bca3fb" containerName="istio-proxy" Oct 30 14:13:19.543169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542512 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" Oct 30 14:13:19.543169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.542528 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7518f76-e056-4ce2-8462-f693ad20cedf" containerName="revoke-keys" Oct 30 14:13:19.547356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.547333 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.564922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.564829 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq"] Oct 30 14:13:19.569334 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569304 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.569457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569365 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.569457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569441 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.569535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569523 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65jz\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-kube-api-access-p65jz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.569600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569580 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.569671 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569634 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.569729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569697 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.569786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569727 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.569903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.569884 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.671087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.671048 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.671314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.671114 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.671314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.671144 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.673139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.672076 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.673139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.673056 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.673412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.673390 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.673495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.671225 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65jz\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-kube-api-access-p65jz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.673554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.673524 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.673609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.673588 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.673695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.673666 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.673750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.673703 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.684065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.677721 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.684065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.677865 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.687359 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.687295 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.687573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.687553 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.690687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.690525 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.699135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.699101 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65jz\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-kube-api-access-p65jz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.701519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.701496 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jvvbq\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:19.873684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:19.873590 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:20.199049 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:13:20.199000 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda307e4c8_46bf_44bb_ba8c_1dbb15871f61.slice/crio-583075c8642a794aceb03f1a569f74c6e1794e03341cde66040e7c381c865ca7 WatchSource:0}: Error finding container 583075c8642a794aceb03f1a569f74c6e1794e03341cde66040e7c381c865ca7: Status 404 returned error can't find the container with id 583075c8642a794aceb03f1a569f74c6e1794e03341cde66040e7c381c865ca7 Oct 30 14:13:20.200767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:20.200740 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq"] Oct 30 14:13:20.201831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:20.201794 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:13:20.201900 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:20.201868 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:13:20.201941 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:20.201897 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:13:20.584676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:20.584585 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" event={"ID":"a307e4c8-46bf-44bb-ba8c-1dbb15871f61","Type":"ContainerStarted","Data":"81a3fd96841822e96f44771cc22a0d01b8c801efe057f9c0359c2bf1fb79278a"} Oct 30 14:13:20.584676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:20.584648 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" event={"ID":"a307e4c8-46bf-44bb-ba8c-1dbb15871f61","Type":"ContainerStarted","Data":"583075c8642a794aceb03f1a569f74c6e1794e03341cde66040e7c381c865ca7"} Oct 30 14:13:20.656393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:20.656342 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podStartSLOduration=1.656329012 podStartE2EDuration="1.656329012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:20.651032505 +0000 UTC m=+1430.451537950" watchObservedRunningTime="2026-10-30 14:13:20.656329012 +0000 UTC m=+1430.456834455" Oct 30 14:13:20.874602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:20.874525 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:21.519969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:21.519932 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9"] Oct 30 14:13:21.520238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:21.520208 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" podUID="e613965e-4ed2-4cb6-812f-e2d6596aa4a3" containerName="maas-api" containerID="cri-o://f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653" gracePeriod=30 Oct 30 14:13:21.902237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:21.898327 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 14:13:21.902237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:21.898433 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 14:13:21.995104 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:21.995075 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:13:22.021141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.021106 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-maas-api-tls\") pod \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " Oct 30 14:13:22.021341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.021217 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-metrics-certs\") pod \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " Oct 30 14:13:22.021341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.021276 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-kube-api-access-w9sf5\" (UniqueName: \"kubernetes.io/projected/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-kube-api-access-w9sf5\") pod \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " Oct 30 14:13:22.021440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.021383 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-openshift-service-ca\") pod \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\" (UID: \"e613965e-4ed2-4cb6-812f-e2d6596aa4a3\") " Oct 30 14:13:22.021787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.021760 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-openshift-service-ca" pod "e613965e-4ed2-4cb6-812f-e2d6596aa4a3" (UID: "e613965e-4ed2-4cb6-812f-e2d6596aa4a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:22.022694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.022669 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.024006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.023618 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-maas-api-tls" pod "e613965e-4ed2-4cb6-812f-e2d6596aa4a3" (UID: "e613965e-4ed2-4cb6-812f-e2d6596aa4a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:22.025704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.025628 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-metrics-certs" pod "e613965e-4ed2-4cb6-812f-e2d6596aa4a3" (UID: "e613965e-4ed2-4cb6-812f-e2d6596aa4a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:13:22.025927 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.025901 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-kube-api-access-w9sf5" pod "e613965e-4ed2-4cb6-812f-e2d6596aa4a3" (UID: "e613965e-4ed2-4cb6-812f-e2d6596aa4a3"). InnerVolumeSpecName "kube-api-access-w9sf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:22.126126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.125224 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.126126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.125264 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.126126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.125280 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9sf5\" (UniqueName: \"kubernetes.io/projected/e613965e-4ed2-4cb6-812f-e2d6596aa4a3-kube-api-access-w9sf5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:22.621324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.621280 5824 generic.go:356] "Generic (PLEG): container finished" podID="e613965e-4ed2-4cb6-812f-e2d6596aa4a3" containerID="f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653" exitCode=0 Oct 30 14:13:22.622306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.622274 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" event={"ID":"e613965e-4ed2-4cb6-812f-e2d6596aa4a3","Type":"ContainerDied","Data":"f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653"} Oct 30 14:13:22.622458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.622309 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" Oct 30 14:13:22.622458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.622349 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9" event={"ID":"e613965e-4ed2-4cb6-812f-e2d6596aa4a3","Type":"ContainerDied","Data":"81a96d013b547a38443e263785d88ae669d5b0362b6602ccc77b42779cbdf7be"} Oct 30 14:13:22.622458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.622375 5824 scope.go:122] "RemoveContainer" containerID="f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653" Oct 30 14:13:22.634964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.634794 5824 scope.go:122] "RemoveContainer" containerID="f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653" Oct 30 14:13:22.635252 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:13:22.635219 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653\": container with ID starting with f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653 not found: ID does not exist" containerID="f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653" Oct 30 14:13:22.635364 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.635258 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653"} err="failed to get container status \"f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653\": rpc error: code = NotFound desc = could not find container \"f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653\": container with ID starting with f65ef3e87285dd8ca2f92c3105001abf8d03d4349291451e0e4b0cbdd23b9653 not found: ID does not exist" Oct 30 14:13:22.685766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.685142 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9"] Oct 30 14:13:22.698127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.698097 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-da21809f-f6fc89968-jw8g9"] Oct 30 14:13:22.875320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.874931 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 14:13:22.875320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:22.875045 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 14:13:23.879210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:23.877520 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 14:13:23.879210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:23.877582 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 14:13:24.940730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:24.940685 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 14:13:24.941282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:24.940766 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 14:13:24.992218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:24.985316 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.030217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.014700 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e613965e-4ed2-4cb6-812f-e2d6596aa4a3" path="/var/lib/kubelet/pods/e613965e-4ed2-4cb6-812f-e2d6596aa4a3/volumes" Oct 30 14:13:25.346663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.346562 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.415744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.414310 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.506675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.506618 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.508554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.508506 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.580245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.506550 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 14:13:25.615338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.615236 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.639392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.637271 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 14:13:25.899268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.895003 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 14:13:25.899268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.895074 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 14:13:25.920989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.920943 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 30 14:13:25.927585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.927257 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.988610 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.986277 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:25.988610 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:25.986824 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:13:26.045214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.043580 5824 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:26.378218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.377469 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 14:13:26.415213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.414681 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 14:13:26.440286 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.440246 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:13:26.440485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.440374 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:13:26.536801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.536729 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:13:26.565213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.541274 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:26.565213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.544774 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:13:26.730150 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.730117 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:13:26.755703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.755675 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:13:26.885142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.885104 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 14:13:26.885434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.885400 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 14:13:26.925581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:26.917749 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:13:27.234378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:27.234345 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:13:27.234886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:27.234514 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:13:27.874554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:27.874522 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 14:13:27.874735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:27.874575 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 14:13:29.742009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:29.741974 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:29.742792 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:29.742760 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:29.744625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:29.744605 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:13:30.630370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:30.630333 5824 scope.go:122] "RemoveContainer" containerID="fb4608ef495921b27e2b02b8730c4893ef64b5551cc6b50a73b771997e872a36" Oct 30 14:13:30.642648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:30.642622 5824 scope.go:122] "RemoveContainer" containerID="52afe3b07dc1b7893b65668631b54f425c4e7bcb95cd87d2c14254823268c5c4" Oct 30 14:13:30.652940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:30.652915 5824 scope.go:122] "RemoveContainer" containerID="1d6d2c48f15c081b14e768fbd7a8f02fe9a11ee642521408e287301886020501" Oct 30 14:13:33.642234 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:13:33.642177 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:13:37.570271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.570230 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf"] Oct 30 14:13:37.571285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.571265 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e613965e-4ed2-4cb6-812f-e2d6596aa4a3" containerName="maas-api" Oct 30 14:13:37.571285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.571283 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e613965e-4ed2-4cb6-812f-e2d6596aa4a3" containerName="maas-api" Oct 30 14:13:37.571417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.571395 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6ba0ae-bf20-43b5-9dd0-d27882b9b7e0" containerName="istio-proxy" Oct 30 14:13:37.571417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.571408 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e613965e-4ed2-4cb6-812f-e2d6596aa4a3" containerName="maas-api" Oct 30 14:13:37.577407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.577388 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:37.592560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.592516 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf"] Oct 30 14:13:37.601387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.601360 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4522b19c\"" Oct 30 14:13:37.601654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.601513 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4522b19c-dockercfg-5mfn4\"" Oct 30 14:13:37.623833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.623805 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh"] Oct 30 14:13:37.675805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.675771 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh"] Oct 30 14:13:37.675995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.675891 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:37.802214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.801630 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca3d79e-1078-4ac2-8025-7c6a97773839-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf\" (UID: \"bca3d79e-1078-4ac2-8025-7c6a97773839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:37.802214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.801906 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjmh\" (UniqueName: \"kubernetes.io/projected/bca3d79e-1078-4ac2-8025-7c6a97773839-kube-api-access-nwjmh\") pod \"payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf\" (UID: \"bca3d79e-1078-4ac2-8025-7c6a97773839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:37.903477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.903384 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjmh\" (UniqueName: \"kubernetes.io/projected/bca3d79e-1078-4ac2-8025-7c6a97773839-kube-api-access-nwjmh\") pod \"payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf\" (UID: \"bca3d79e-1078-4ac2-8025-7c6a97773839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:37.903626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.903518 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4xw\" (UniqueName: \"kubernetes.io/projected/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-kube-api-access-7f4xw\") pod \"payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh\" (UID: \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\") " pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:37.903626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.903605 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca3d79e-1078-4ac2-8025-7c6a97773839-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf\" (UID: \"bca3d79e-1078-4ac2-8025-7c6a97773839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:37.903883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.903856 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh\" (UID: \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\") " pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:37.904326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.904308 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca3d79e-1078-4ac2-8025-7c6a97773839-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf\" (UID: \"bca3d79e-1078-4ac2-8025-7c6a97773839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:37.931684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:37.931655 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjmh\" (UniqueName: \"kubernetes.io/projected/bca3d79e-1078-4ac2-8025-7c6a97773839-kube-api-access-nwjmh\") pod \"payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf\" (UID: \"bca3d79e-1078-4ac2-8025-7c6a97773839\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:38.005275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:38.005234 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh\" (UID: \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\") " pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:38.005457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:38.005402 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4xw\" (UniqueName: \"kubernetes.io/projected/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-kube-api-access-7f4xw\") pod \"payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh\" (UID: \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\") " pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:38.005951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:38.005929 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh\" (UID: \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\") " pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:38.033884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:38.033860 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4xw\" (UniqueName: \"kubernetes.io/projected/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-kube-api-access-7f4xw\") pod \"payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh\" (UID: \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\") " pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:38.151326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:38.151292 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:38.195652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:38.194609 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:38.503735 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:13:38.503330 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dd7dc4_05ea_4cef_bf64_0c48fd26fca3.slice/crio-26228eaf76be89ecd94ab4076ba835c907b6ef44fbb1ddc9e8ef6c61f1ac4e7f WatchSource:0}: Error finding container 26228eaf76be89ecd94ab4076ba835c907b6ef44fbb1ddc9e8ef6c61f1ac4e7f: Status 404 returned error can't find the container with id 26228eaf76be89ecd94ab4076ba835c907b6ef44fbb1ddc9e8ef6c61f1ac4e7f Oct 30 14:13:38.509373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:38.509347 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh"] Oct 30 14:13:38.567536 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:13:38.567508 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca3d79e_1078_4ac2_8025_7c6a97773839.slice/crio-8f8800481e2fb42948b427eaf22212524131c96fb484ed45000ac13affa00f3c WatchSource:0}: Error finding container 8f8800481e2fb42948b427eaf22212524131c96fb484ed45000ac13affa00f3c: Status 404 returned error can't find the container with id 8f8800481e2fb42948b427eaf22212524131c96fb484ed45000ac13affa00f3c Oct 30 14:13:38.570320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:38.570294 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf"] Oct 30 14:13:39.361575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:39.361538 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" event={"ID":"bca3d79e-1078-4ac2-8025-7c6a97773839","Type":"ContainerStarted","Data":"db690182209815b99799d574df8a2646e052c5442e3408a8512b7a7ef173ae3b"} Oct 30 14:13:39.361739 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:39.361605 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" event={"ID":"bca3d79e-1078-4ac2-8025-7c6a97773839","Type":"ContainerStarted","Data":"8f8800481e2fb42948b427eaf22212524131c96fb484ed45000ac13affa00f3c"} Oct 30 14:13:39.361821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:39.361755 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:39.363331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:39.363307 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" event={"ID":"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3","Type":"ContainerStarted","Data":"60016c2d93b157dfd049d0f6419a56f0f986bc05a8de8d6b936fc3ffff71109c"} Oct 30 14:13:39.363466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:39.363349 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" event={"ID":"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3","Type":"ContainerStarted","Data":"26228eaf76be89ecd94ab4076ba835c907b6ef44fbb1ddc9e8ef6c61f1ac4e7f"} Oct 30 14:13:39.363466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:39.363381 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:39.409944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:39.409891 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" podStartSLOduration=2.409879074 podStartE2EDuration="2.409879074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:39.405747362 +0000 UTC m=+1449.206252808" watchObservedRunningTime="2026-10-30 14:13:39.409879074 +0000 UTC m=+1449.210384517" Oct 30 14:13:39.458107 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:39.458066 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" podStartSLOduration=2.458052788 podStartE2EDuration="2.458052788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:39.456708935 +0000 UTC m=+1449.257214380" watchObservedRunningTime="2026-10-30 14:13:39.458052788 +0000 UTC m=+1449.258558231" Oct 30 14:13:44.259268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:44.259171 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5"] Oct 30 14:13:44.305011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:44.304971 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb"] Oct 30 14:13:49.259899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.259858 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" podUID="41043006-3e00-4724-bef9-415bb5f95c57" containerName="payload-processing" containerID="cri-o://33c352ba08ac1217fd42db9155ff943726151e19272eb03c488230fe1879822d" gracePeriod=25 Oct 30 14:13:49.305473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.305439 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" podUID="4c99baa2-49ef-4912-bf3d-5f8c613398a5" containerName="payload-pre-processing" containerID="cri-o://1f3c691db510263a5fafd22fcdbda41880b5457ef1f985df730141ed3f75dff8" gracePeriod=25 Oct 30 14:13:49.457978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.457944 5824 generic.go:356] "Generic (PLEG): container finished" podID="4c99baa2-49ef-4912-bf3d-5f8c613398a5" containerID="1f3c691db510263a5fafd22fcdbda41880b5457ef1f985df730141ed3f75dff8" exitCode=0 Oct 30 14:13:49.458132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.458073 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" event={"ID":"4c99baa2-49ef-4912-bf3d-5f8c613398a5","Type":"ContainerDied","Data":"1f3c691db510263a5fafd22fcdbda41880b5457ef1f985df730141ed3f75dff8"} Oct 30 14:13:49.459870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.459774 5824 generic.go:356] "Generic (PLEG): container finished" podID="41043006-3e00-4724-bef9-415bb5f95c57" containerID="33c352ba08ac1217fd42db9155ff943726151e19272eb03c488230fe1879822d" exitCode=0 Oct 30 14:13:49.459870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.459830 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" event={"ID":"41043006-3e00-4724-bef9-415bb5f95c57","Type":"ContainerDied","Data":"33c352ba08ac1217fd42db9155ff943726151e19272eb03c488230fe1879822d"} Oct 30 14:13:49.629238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.629177 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:13:49.662628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.662601 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:13:49.718847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.718812 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41043006-3e00-4724-bef9-415bb5f95c57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41043006-3e00-4724-bef9-415bb5f95c57-plugins-config-volume\") pod \"41043006-3e00-4724-bef9-415bb5f95c57\" (UID: \"41043006-3e00-4724-bef9-415bb5f95c57\") " Oct 30 14:13:49.719120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.719102 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41043006-3e00-4724-bef9-415bb5f95c57-kube-api-access-xngqm\" (UniqueName: \"kubernetes.io/projected/41043006-3e00-4724-bef9-415bb5f95c57-kube-api-access-xngqm\") pod \"41043006-3e00-4724-bef9-415bb5f95c57\" (UID: \"41043006-3e00-4724-bef9-415bb5f95c57\") " Oct 30 14:13:49.719357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.719291 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41043006-3e00-4724-bef9-415bb5f95c57-plugins-config-volume" pod "41043006-3e00-4724-bef9-415bb5f95c57" (UID: "41043006-3e00-4724-bef9-415bb5f95c57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:49.719971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.719945 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41043006-3e00-4724-bef9-415bb5f95c57-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:49.721561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.721532 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41043006-3e00-4724-bef9-415bb5f95c57-kube-api-access-xngqm" pod "41043006-3e00-4724-bef9-415bb5f95c57" (UID: "41043006-3e00-4724-bef9-415bb5f95c57"). InnerVolumeSpecName "kube-api-access-xngqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:49.820799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.820694 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c99baa2-49ef-4912-bf3d-5f8c613398a5-kube-api-access-vdnm4\" (UniqueName: \"kubernetes.io/projected/4c99baa2-49ef-4912-bf3d-5f8c613398a5-kube-api-access-vdnm4\") pod \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\" (UID: \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\") " Oct 30 14:13:49.820984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.820898 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c99baa2-49ef-4912-bf3d-5f8c613398a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c99baa2-49ef-4912-bf3d-5f8c613398a5-plugins-config-volume\") pod \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\" (UID: \"4c99baa2-49ef-4912-bf3d-5f8c613398a5\") " Oct 30 14:13:49.821375 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.821340 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c99baa2-49ef-4912-bf3d-5f8c613398a5-plugins-config-volume" pod "4c99baa2-49ef-4912-bf3d-5f8c613398a5" (UID: "4c99baa2-49ef-4912-bf3d-5f8c613398a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:13:49.821648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.821620 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c99baa2-49ef-4912-bf3d-5f8c613398a5-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:49.821648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.821650 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngqm\" (UniqueName: \"kubernetes.io/projected/41043006-3e00-4724-bef9-415bb5f95c57-kube-api-access-xngqm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:49.823357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.823328 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c99baa2-49ef-4912-bf3d-5f8c613398a5-kube-api-access-vdnm4" pod "4c99baa2-49ef-4912-bf3d-5f8c613398a5" (UID: "4c99baa2-49ef-4912-bf3d-5f8c613398a5"). InnerVolumeSpecName "kube-api-access-vdnm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:13:49.927123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:49.927088 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdnm4\" (UniqueName: \"kubernetes.io/projected/4c99baa2-49ef-4912-bf3d-5f8c613398a5-kube-api-access-vdnm4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:13:50.374542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.374488 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:13:50.375163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.374714 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:13:50.478240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.478206 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" event={"ID":"41043006-3e00-4724-bef9-415bb5f95c57","Type":"ContainerDied","Data":"e99e590ffd92421b86488afbed3d135cface4f4d5a63b308cebb7c654a7c563c"} Oct 30 14:13:50.478539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.478520 5824 scope.go:122] "RemoveContainer" containerID="33c352ba08ac1217fd42db9155ff943726151e19272eb03c488230fe1879822d" Oct 30 14:13:50.478709 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.478258 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5" Oct 30 14:13:50.481343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.480609 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" event={"ID":"4c99baa2-49ef-4912-bf3d-5f8c613398a5","Type":"ContainerDied","Data":"4753030f5b9f637b92ae52eda488b739c3d0f1043a20ce3725b0507494a0923f"} Oct 30 14:13:50.481343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.480706 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb" Oct 30 14:13:50.494975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.494954 5824 scope.go:122] "RemoveContainer" containerID="1f3c691db510263a5fafd22fcdbda41880b5457ef1f985df730141ed3f75dff8" Oct 30 14:13:50.545280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.545219 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5"] Oct 30 14:13:50.552994 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.552970 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-25917498-5b656767dd-2xdq5"] Oct 30 14:13:50.591096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.591042 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb"] Oct 30 14:13:50.596148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:50.596123 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-25917498-769cb747f9-hhxfb"] Oct 30 14:13:52.317827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:52.317796 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41043006-3e00-4724-bef9-415bb5f95c57" path="/var/lib/kubelet/pods/41043006-3e00-4724-bef9-415bb5f95c57/volumes" Oct 30 14:13:52.318440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:52.318417 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c99baa2-49ef-4912-bf3d-5f8c613398a5" path="/var/lib/kubelet/pods/4c99baa2-49ef-4912-bf3d-5f8c613398a5/volumes" Oct 30 14:13:58.077072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:58.077030 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d"] Oct 30 14:13:58.102990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:58.102962 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0590961f-n7n6d"] Oct 30 14:13:58.318749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:13:58.318714 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ec0077-ba94-4422-83eb-fc127f893b07" path="/var/lib/kubelet/pods/90ec0077-ba94-4422-83eb-fc127f893b07/volumes" Oct 30 14:14:03.642299 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:14:03.642261 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:14:14.932127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.932092 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9"] Oct 30 14:14:14.932921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.932903 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4c99baa2-49ef-4912-bf3d-5f8c613398a5" containerName="payload-pre-processing" Oct 30 14:14:14.932984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.932922 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c99baa2-49ef-4912-bf3d-5f8c613398a5" containerName="payload-pre-processing" Oct 30 14:14:14.932984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.932957 5824 cpu_manager.go:414] "containerMap: removing container" podUID="41043006-3e00-4724-bef9-415bb5f95c57" containerName="payload-processing" Oct 30 14:14:14.932984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.932963 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41043006-3e00-4724-bef9-415bb5f95c57" containerName="payload-processing" Oct 30 14:14:14.933083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.933064 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="41043006-3e00-4724-bef9-415bb5f95c57" containerName="payload-processing" Oct 30 14:14:14.933083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.933079 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c99baa2-49ef-4912-bf3d-5f8c613398a5" containerName="payload-pre-processing" Oct 30 14:14:14.938504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.938486 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:14.966332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.966301 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9"] Oct 30 14:14:14.972977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.972952 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:14:14.973115 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:14.973016 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:14:15.044624 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.044593 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7nd\" (UniqueName: \"kubernetes.io/projected/8d918c2d-a799-485e-9b3d-92f2e73b68fe-kube-api-access-rj7nd\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.044803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.044782 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.044935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.044921 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.146575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.146546 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.146698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.146652 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.146753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.146740 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7nd\" (UniqueName: \"kubernetes.io/projected/8d918c2d-a799-485e-9b3d-92f2e73b68fe-kube-api-access-rj7nd\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.147285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.147224 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.148937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.148919 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.175383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.175356 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7nd\" (UniqueName: \"kubernetes.io/projected/8d918c2d-a799-485e-9b3d-92f2e73b68fe-kube-api-access-rj7nd\") pod \"maas-api-revoke-sub-fe66b9bc-rgtr9\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.254999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.254939 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:15.597395 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:14:15.597365 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d918c2d_a799_485e_9b3d_92f2e73b68fe.slice/crio-369179d1e93d9a47de35440f6849052f62390c6e73686846982e3429ae35494a WatchSource:0}: Error finding container 369179d1e93d9a47de35440f6849052f62390c6e73686846982e3429ae35494a: Status 404 returned error can't find the container with id 369179d1e93d9a47de35440f6849052f62390c6e73686846982e3429ae35494a Oct 30 14:14:15.600935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.600909 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9"] Oct 30 14:14:15.723010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:15.722973 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" event={"ID":"8d918c2d-a799-485e-9b3d-92f2e73b68fe","Type":"ContainerStarted","Data":"369179d1e93d9a47de35440f6849052f62390c6e73686846982e3429ae35494a"} Oct 30 14:14:16.736460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:16.736350 5824 generic.go:356] "Generic (PLEG): container finished" podID="8d918c2d-a799-485e-9b3d-92f2e73b68fe" containerID="ec027e5b69a387d563a44d27606a97fdb3656b50d615c536ad6b4056ce2bc023" exitCode=0 Oct 30 14:14:16.736460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:16.736419 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" event={"ID":"8d918c2d-a799-485e-9b3d-92f2e73b68fe","Type":"ContainerDied","Data":"ec027e5b69a387d563a44d27606a97fdb3656b50d615c536ad6b4056ce2bc023"} Oct 30 14:14:17.985352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:17.985318 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:18.116268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.115888 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-openshift-ca\") pod \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " Oct 30 14:14:18.116268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.115981 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-xks-ca\") pod \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " Oct 30 14:14:18.116581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.116344 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d918c2d-a799-485e-9b3d-92f2e73b68fe-kube-api-access-rj7nd\" (UniqueName: \"kubernetes.io/projected/8d918c2d-a799-485e-9b3d-92f2e73b68fe-kube-api-access-rj7nd\") pod \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\" (UID: \"8d918c2d-a799-485e-9b3d-92f2e73b68fe\") " Oct 30 14:14:18.161170 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.161133 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-openshift-ca" pod "8d918c2d-a799-485e-9b3d-92f2e73b68fe" (UID: "8d918c2d-a799-485e-9b3d-92f2e73b68fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:14:18.162291 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.161768 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:18.168130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.168090 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-xks-ca" pod "8d918c2d-a799-485e-9b3d-92f2e73b68fe" (UID: "8d918c2d-a799-485e-9b3d-92f2e73b68fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:14:18.170951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.170920 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d918c2d-a799-485e-9b3d-92f2e73b68fe-kube-api-access-rj7nd" pod "8d918c2d-a799-485e-9b3d-92f2e73b68fe" (UID: "8d918c2d-a799-485e-9b3d-92f2e73b68fe"). InnerVolumeSpecName "kube-api-access-rj7nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:14:18.263576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.263536 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7nd\" (UniqueName: \"kubernetes.io/projected/8d918c2d-a799-485e-9b3d-92f2e73b68fe-kube-api-access-rj7nd\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:18.263576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.263575 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d918c2d-a799-485e-9b3d-92f2e73b68fe-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:18.763113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.763077 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" Oct 30 14:14:18.763283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.763082 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9" event={"ID":"8d918c2d-a799-485e-9b3d-92f2e73b68fe","Type":"ContainerDied","Data":"369179d1e93d9a47de35440f6849052f62390c6e73686846982e3429ae35494a"} Oct 30 14:14:18.763283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:18.763176 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369179d1e93d9a47de35440f6849052f62390c6e73686846982e3429ae35494a" Oct 30 14:14:22.065566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.065533 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j"] Oct 30 14:14:22.078291 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.078265 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4535894a-rkc2j"] Oct 30 14:14:22.319489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.319407 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9387165a-731d-44fc-a638-203ed61da74f" path="/var/lib/kubelet/pods/9387165a-731d-44fc-a638-203ed61da74f/volumes" Oct 30 14:14:22.632274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.632174 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659"] Oct 30 14:14:22.633199 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.633157 5824 cpu_manager.go:414] "containerMap: removing container" podUID="8d918c2d-a799-485e-9b3d-92f2e73b68fe" containerName="revoke-keys" Oct 30 14:14:22.633199 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.633180 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d918c2d-a799-485e-9b3d-92f2e73b68fe" containerName="revoke-keys" Oct 30 14:14:22.633365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.633352 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d918c2d-a799-485e-9b3d-92f2e73b68fe" containerName="revoke-keys" Oct 30 14:14:22.637147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.637128 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.657126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.657102 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:14:22.657273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.657224 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:14:22.663361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.663334 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659"] Oct 30 14:14:22.733930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.733890 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.734129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.734072 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzg9v\" (UniqueName: \"kubernetes.io/projected/e347019e-4fdb-4bd5-a51d-13e08cee3e21-kube-api-access-dzg9v\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.734227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.734158 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.842553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.842508 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.854250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.848888 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.854250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.849715 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.854250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.849945 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzg9v\" (UniqueName: \"kubernetes.io/projected/e347019e-4fdb-4bd5-a51d-13e08cee3e21-kube-api-access-dzg9v\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.892976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.892884 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.894767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.894744 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzg9v\" (UniqueName: \"kubernetes.io/projected/e347019e-4fdb-4bd5-a51d-13e08cee3e21-kube-api-access-dzg9v\") pod \"maas-api-revoke-keys-e2e-ait-4522b19c-zt659\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:22.953853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:22.953826 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:23.275547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:23.275510 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659"] Oct 30 14:14:23.889056 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:23.888973 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" event={"ID":"e347019e-4fdb-4bd5-a51d-13e08cee3e21","Type":"ContainerStarted","Data":"0cb69832aecc25d224a86198075e1f20549581168c802585f3d8737a4f5068e4"} Oct 30 14:14:23.889056 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:23.889036 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" event={"ID":"e347019e-4fdb-4bd5-a51d-13e08cee3e21","Type":"ContainerStarted","Data":"342ce84b8135945bc894c7322476369d1b0f623063d71d730920e40a474336de"} Oct 30 14:14:23.939567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:23.939521 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" podStartSLOduration=1.939507111 podStartE2EDuration="1.939507111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:23.936306051 +0000 UTC m=+1493.736811495" watchObservedRunningTime="2026-10-30 14:14:23.939507111 +0000 UTC m=+1493.740012564" Oct 30 14:14:24.901341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:24.901303 5824 generic.go:356] "Generic (PLEG): container finished" podID="e347019e-4fdb-4bd5-a51d-13e08cee3e21" containerID="0cb69832aecc25d224a86198075e1f20549581168c802585f3d8737a4f5068e4" exitCode=0 Oct 30 14:14:24.902379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:24.901509 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" event={"ID":"e347019e-4fdb-4bd5-a51d-13e08cee3e21","Type":"ContainerDied","Data":"0cb69832aecc25d224a86198075e1f20549581168c802585f3d8737a4f5068e4"} Oct 30 14:14:26.064691 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.064668 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:26.096521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.096497 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-xks-ca\") pod \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " Oct 30 14:14:26.096662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.096649 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-openshift-ca\") pod \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " Oct 30 14:14:26.096805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.096788 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e347019e-4fdb-4bd5-a51d-13e08cee3e21-kube-api-access-dzg9v\" (UniqueName: \"kubernetes.io/projected/e347019e-4fdb-4bd5-a51d-13e08cee3e21-kube-api-access-dzg9v\") pod \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\" (UID: \"e347019e-4fdb-4bd5-a51d-13e08cee3e21\") " Oct 30 14:14:26.097019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.096995 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-openshift-ca" pod "e347019e-4fdb-4bd5-a51d-13e08cee3e21" (UID: "e347019e-4fdb-4bd5-a51d-13e08cee3e21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:14:26.097440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.097422 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:26.098570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.098542 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-xks-ca" pod "e347019e-4fdb-4bd5-a51d-13e08cee3e21" (UID: "e347019e-4fdb-4bd5-a51d-13e08cee3e21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:14:26.098685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.098668 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e347019e-4fdb-4bd5-a51d-13e08cee3e21-kube-api-access-dzg9v" pod "e347019e-4fdb-4bd5-a51d-13e08cee3e21" (UID: "e347019e-4fdb-4bd5-a51d-13e08cee3e21"). InnerVolumeSpecName "kube-api-access-dzg9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:14:26.199391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.199364 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e347019e-4fdb-4bd5-a51d-13e08cee3e21-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:26.199391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.199390 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzg9v\" (UniqueName: \"kubernetes.io/projected/e347019e-4fdb-4bd5-a51d-13e08cee3e21-kube-api-access-dzg9v\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:26.930401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.930347 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" event={"ID":"e347019e-4fdb-4bd5-a51d-13e08cee3e21","Type":"ContainerDied","Data":"342ce84b8135945bc894c7322476369d1b0f623063d71d730920e40a474336de"} Oct 30 14:14:26.930658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.930523 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342ce84b8135945bc894c7322476369d1b0f623063d71d730920e40a474336de" Oct 30 14:14:26.930811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:26.930684 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659" Oct 30 14:14:30.710579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:30.710541 5824 scope.go:122] "RemoveContainer" containerID="90c72a297487c73051e0a598d7c1b1895360fe4aedae7c1c8baf6850e36d6226" Oct 30 14:14:31.002666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.002581 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl"] Oct 30 14:14:31.003365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.003349 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e347019e-4fdb-4bd5-a51d-13e08cee3e21" containerName="revoke-keys" Oct 30 14:14:31.003427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.003366 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e347019e-4fdb-4bd5-a51d-13e08cee3e21" containerName="revoke-keys" Oct 30 14:14:31.003495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.003486 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e347019e-4fdb-4bd5-a51d-13e08cee3e21" containerName="revoke-keys" Oct 30 14:14:31.008704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.008686 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:31.032353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.032319 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl"] Oct 30 14:14:31.036894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.036865 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9dba4af9-dockercfg-r7jw7\"" Oct 30 14:14:31.036894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.036881 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9dba4af9\"" Oct 30 14:14:31.096372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.086299 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de68d4e-71ca-49be-ba3f-0ece4545245e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl\" (UID: \"9de68d4e-71ca-49be-ba3f-0ece4545245e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:31.096580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.096550 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qslt9\" (UniqueName: \"kubernetes.io/projected/9de68d4e-71ca-49be-ba3f-0ece4545245e-kube-api-access-qslt9\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl\" (UID: \"9de68d4e-71ca-49be-ba3f-0ece4545245e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:31.236680 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.236637 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de68d4e-71ca-49be-ba3f-0ece4545245e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl\" (UID: \"9de68d4e-71ca-49be-ba3f-0ece4545245e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:31.236958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.236788 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qslt9\" (UniqueName: \"kubernetes.io/projected/9de68d4e-71ca-49be-ba3f-0ece4545245e-kube-api-access-qslt9\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl\" (UID: \"9de68d4e-71ca-49be-ba3f-0ece4545245e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:31.237854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.237827 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de68d4e-71ca-49be-ba3f-0ece4545245e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl\" (UID: \"9de68d4e-71ca-49be-ba3f-0ece4545245e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:31.268005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.267934 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qslt9\" (UniqueName: \"kubernetes.io/projected/9de68d4e-71ca-49be-ba3f-0ece4545245e-kube-api-access-qslt9\") pod \"payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl\" (UID: \"9de68d4e-71ca-49be-ba3f-0ece4545245e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:31.323672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.323635 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:31.652797 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:14:31.652766 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de68d4e_71ca_49be_ba3f_0ece4545245e.slice/crio-550d3419c4868d48791b2caa94b28e845a880e70dcad99eee32d151a1e023447 WatchSource:0}: Error finding container 550d3419c4868d48791b2caa94b28e845a880e70dcad99eee32d151a1e023447: Status 404 returned error can't find the container with id 550d3419c4868d48791b2caa94b28e845a880e70dcad99eee32d151a1e023447 Oct 30 14:14:31.655767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.655736 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl"] Oct 30 14:14:31.971377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.971344 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" event={"ID":"9de68d4e-71ca-49be-ba3f-0ece4545245e","Type":"ContainerStarted","Data":"a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7"} Oct 30 14:14:31.971827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.971393 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" event={"ID":"9de68d4e-71ca-49be-ba3f-0ece4545245e","Type":"ContainerStarted","Data":"550d3419c4868d48791b2caa94b28e845a880e70dcad99eee32d151a1e023447"} Oct 30 14:14:31.971827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:31.971570 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:32.016747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:32.016698 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" podStartSLOduration=2.016684318 podStartE2EDuration="2.016684318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:32.015944485 +0000 UTC m=+1501.816449928" watchObservedRunningTime="2026-10-30 14:14:32.016684318 +0000 UTC m=+1501.817189761" Oct 30 14:14:32.855036 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:32.854994 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh"] Oct 30 14:14:32.887799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:32.887762 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf"] Oct 30 14:14:33.646966 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:14:33.646147 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:14:35.081807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:35.081775 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d"] Oct 30 14:14:35.094859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:35.094826 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa74430-sp54d"] Oct 30 14:14:36.319958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:36.319914 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f86e69-b76f-4c1e-a145-578dd368f5b5" path="/var/lib/kubelet/pods/c8f86e69-b76f-4c1e-a145-578dd368f5b5/volumes" Oct 30 14:14:37.856255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:37.856177 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" podUID="89dd7dc4-05ea-4cef-bf64-0c48fd26fca3" containerName="payload-processing" containerID="cri-o://60016c2d93b157dfd049d0f6419a56f0f986bc05a8de8d6b936fc3ffff71109c" gracePeriod=25 Oct 30 14:14:37.889067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:37.889030 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" podUID="bca3d79e-1078-4ac2-8025-7c6a97773839" containerName="payload-pre-processing" containerID="cri-o://db690182209815b99799d574df8a2646e052c5442e3408a8512b7a7ef173ae3b" gracePeriod=25 Oct 30 14:14:38.035163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.035124 5824 generic.go:356] "Generic (PLEG): container finished" podID="bca3d79e-1078-4ac2-8025-7c6a97773839" containerID="db690182209815b99799d574df8a2646e052c5442e3408a8512b7a7ef173ae3b" exitCode=0 Oct 30 14:14:38.035476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.035448 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" event={"ID":"bca3d79e-1078-4ac2-8025-7c6a97773839","Type":"ContainerDied","Data":"db690182209815b99799d574df8a2646e052c5442e3408a8512b7a7ef173ae3b"} Oct 30 14:14:38.038356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.038322 5824 generic.go:356] "Generic (PLEG): container finished" podID="89dd7dc4-05ea-4cef-bf64-0c48fd26fca3" containerID="60016c2d93b157dfd049d0f6419a56f0f986bc05a8de8d6b936fc3ffff71109c" exitCode=0 Oct 30 14:14:38.038462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.038435 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" event={"ID":"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3","Type":"ContainerDied","Data":"60016c2d93b157dfd049d0f6419a56f0f986bc05a8de8d6b936fc3ffff71109c"} Oct 30 14:14:38.282464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.280808 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:14:38.339249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.339181 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:14:38.368943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.368903 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca3d79e-1078-4ac2-8025-7c6a97773839-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca3d79e-1078-4ac2-8025-7c6a97773839-plugins-config-volume\") pod \"bca3d79e-1078-4ac2-8025-7c6a97773839\" (UID: \"bca3d79e-1078-4ac2-8025-7c6a97773839\") " Oct 30 14:14:38.369128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.369046 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-kube-api-access-7f4xw\" (UniqueName: \"kubernetes.io/projected/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-kube-api-access-7f4xw\") pod \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\" (UID: \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\") " Oct 30 14:14:38.369211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.369159 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca3d79e-1078-4ac2-8025-7c6a97773839-kube-api-access-nwjmh\" (UniqueName: \"kubernetes.io/projected/bca3d79e-1078-4ac2-8025-7c6a97773839-kube-api-access-nwjmh\") pod \"bca3d79e-1078-4ac2-8025-7c6a97773839\" (UID: \"bca3d79e-1078-4ac2-8025-7c6a97773839\") " Oct 30 14:14:38.369300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.369275 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-plugins-config-volume\") pod \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\" (UID: \"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3\") " Oct 30 14:14:38.369443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.369368 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca3d79e-1078-4ac2-8025-7c6a97773839-plugins-config-volume" pod "bca3d79e-1078-4ac2-8025-7c6a97773839" (UID: "bca3d79e-1078-4ac2-8025-7c6a97773839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:14:38.369731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.369708 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-plugins-config-volume" pod "89dd7dc4-05ea-4cef-bf64-0c48fd26fca3" (UID: "89dd7dc4-05ea-4cef-bf64-0c48fd26fca3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:14:38.370820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.370789 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca3d79e-1078-4ac2-8025-7c6a97773839-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:38.370913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.370823 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:38.372002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.371981 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca3d79e-1078-4ac2-8025-7c6a97773839-kube-api-access-nwjmh" pod "bca3d79e-1078-4ac2-8025-7c6a97773839" (UID: "bca3d79e-1078-4ac2-8025-7c6a97773839"). InnerVolumeSpecName "kube-api-access-nwjmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:14:38.372150 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.372134 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-kube-api-access-7f4xw" pod "89dd7dc4-05ea-4cef-bf64-0c48fd26fca3" (UID: "89dd7dc4-05ea-4cef-bf64-0c48fd26fca3"). InnerVolumeSpecName "kube-api-access-7f4xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:14:38.474250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.472850 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwjmh\" (UniqueName: \"kubernetes.io/projected/bca3d79e-1078-4ac2-8025-7c6a97773839-kube-api-access-nwjmh\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:38.474250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:38.472882 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4xw\" (UniqueName: \"kubernetes.io/projected/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3-kube-api-access-7f4xw\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:14:39.050916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.050881 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" Oct 30 14:14:39.050916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.050879 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf" event={"ID":"bca3d79e-1078-4ac2-8025-7c6a97773839","Type":"ContainerDied","Data":"8f8800481e2fb42948b427eaf22212524131c96fb484ed45000ac13affa00f3c"} Oct 30 14:14:39.051467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.050982 5824 scope.go:122] "RemoveContainer" containerID="db690182209815b99799d574df8a2646e052c5442e3408a8512b7a7ef173ae3b" Oct 30 14:14:39.068427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.057648 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" event={"ID":"89dd7dc4-05ea-4cef-bf64-0c48fd26fca3","Type":"ContainerDied","Data":"26228eaf76be89ecd94ab4076ba835c907b6ef44fbb1ddc9e8ef6c61f1ac4e7f"} Oct 30 14:14:39.068427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.057781 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh" Oct 30 14:14:39.081896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.081869 5824 scope.go:122] "RemoveContainer" containerID="60016c2d93b157dfd049d0f6419a56f0f986bc05a8de8d6b936fc3ffff71109c" Oct 30 14:14:39.133114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.133052 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf"] Oct 30 14:14:39.137030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.137007 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4522b19c-57c8cd98c9-9w8xf"] Oct 30 14:14:39.187832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.187720 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh"] Oct 30 14:14:39.194591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.194558 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4522b19c-b46b9b49d-4ddlh"] Oct 30 14:14:39.198010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.197981 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659"] Oct 30 14:14:39.201442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.201423 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4522b19c-zt659"] Oct 30 14:14:39.330398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.330356 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2"] Oct 30 14:14:39.331121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.331100 5824 cpu_manager.go:414] "containerMap: removing container" podUID="89dd7dc4-05ea-4cef-bf64-0c48fd26fca3" containerName="payload-processing" Oct 30 14:14:39.331284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.331121 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89dd7dc4-05ea-4cef-bf64-0c48fd26fca3" containerName="payload-processing" Oct 30 14:14:39.331284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.331149 5824 cpu_manager.go:414] "containerMap: removing container" podUID="bca3d79e-1078-4ac2-8025-7c6a97773839" containerName="payload-pre-processing" Oct 30 14:14:39.331284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.331157 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca3d79e-1078-4ac2-8025-7c6a97773839" containerName="payload-pre-processing" Oct 30 14:14:39.331462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.331444 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="89dd7dc4-05ea-4cef-bf64-0c48fd26fca3" containerName="payload-processing" Oct 30 14:14:39.331526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.331466 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca3d79e-1078-4ac2-8025-7c6a97773839" containerName="payload-pre-processing" Oct 30 14:14:39.336779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.336760 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:39.354983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.354958 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-0efe43\"" Oct 30 14:14:39.356672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.356647 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2"] Oct 30 14:14:39.361413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.361391 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-0efe43-dockercfg-kn5pg\"" Oct 30 14:14:39.389629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.389455 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjcjr\" (UniqueName: \"kubernetes.io/projected/ae95a746-48ba-4686-8f37-ae30206dc275-kube-api-access-wjcjr\") pod \"payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2\" (UID: \"ae95a746-48ba-4686-8f37-ae30206dc275\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:39.389629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.389531 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae95a746-48ba-4686-8f37-ae30206dc275-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2\" (UID: \"ae95a746-48ba-4686-8f37-ae30206dc275\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:39.416595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.416557 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66"] Oct 30 14:14:39.519317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.505503 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:39.520069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.508105 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjcjr\" (UniqueName: \"kubernetes.io/projected/ae95a746-48ba-4686-8f37-ae30206dc275-kube-api-access-wjcjr\") pod \"payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2\" (UID: \"ae95a746-48ba-4686-8f37-ae30206dc275\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:39.520325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.520306 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae95a746-48ba-4686-8f37-ae30206dc275-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2\" (UID: \"ae95a746-48ba-4686-8f37-ae30206dc275\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:39.521520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.521476 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae95a746-48ba-4686-8f37-ae30206dc275-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2\" (UID: \"ae95a746-48ba-4686-8f37-ae30206dc275\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:39.530237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.525992 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66"] Oct 30 14:14:39.540349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.539508 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjcjr\" (UniqueName: \"kubernetes.io/projected/ae95a746-48ba-4686-8f37-ae30206dc275-kube-api-access-wjcjr\") pod \"payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2\" (UID: \"ae95a746-48ba-4686-8f37-ae30206dc275\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:39.623287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.622528 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66\" (UID: \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:39.623483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.623452 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9dk\" (UniqueName: \"kubernetes.io/projected/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-kube-api-access-bp9dk\") pod \"payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66\" (UID: \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:39.653492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.653394 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:39.726270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.726239 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9dk\" (UniqueName: \"kubernetes.io/projected/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-kube-api-access-bp9dk\") pod \"payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66\" (UID: \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:39.726577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.726479 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66\" (UID: \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:39.727534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.727505 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66\" (UID: \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:39.759322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.759287 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9dk\" (UniqueName: \"kubernetes.io/projected/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-kube-api-access-bp9dk\") pod \"payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66\" (UID: \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:39.843997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.843967 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:39.997535 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:14:39.997483 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae95a746_48ba_4686_8f37_ae30206dc275.slice/crio-c9ef7c74c78429b547673c2bcf8c6f412074f32c070bd804e6858e47f7f905f0 WatchSource:0}: Error finding container c9ef7c74c78429b547673c2bcf8c6f412074f32c070bd804e6858e47f7f905f0: Status 404 returned error can't find the container with id c9ef7c74c78429b547673c2bcf8c6f412074f32c070bd804e6858e47f7f905f0 Oct 30 14:14:39.997736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:39.997707 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2"] Oct 30 14:14:40.068523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:40.068490 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" event={"ID":"ae95a746-48ba-4686-8f37-ae30206dc275","Type":"ContainerStarted","Data":"c9ef7c74c78429b547673c2bcf8c6f412074f32c070bd804e6858e47f7f905f0"} Oct 30 14:14:40.179756 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:14:40.179724 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f31e1a3_877c_476f_bbf8_7eef6fcc3bbf.slice/crio-6cb89222fb3ef242a327f33933fee444f912bbc138f355f83ca6e113339ec6e6 WatchSource:0}: Error finding container 6cb89222fb3ef242a327f33933fee444f912bbc138f355f83ca6e113339ec6e6: Status 404 returned error can't find the container with id 6cb89222fb3ef242a327f33933fee444f912bbc138f355f83ca6e113339ec6e6 Oct 30 14:14:40.181234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:40.181203 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66"] Oct 30 14:14:40.320462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:40.320434 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89dd7dc4-05ea-4cef-bf64-0c48fd26fca3" path="/var/lib/kubelet/pods/89dd7dc4-05ea-4cef-bf64-0c48fd26fca3/volumes" Oct 30 14:14:40.320898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:40.320884 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca3d79e-1078-4ac2-8025-7c6a97773839" path="/var/lib/kubelet/pods/bca3d79e-1078-4ac2-8025-7c6a97773839/volumes" Oct 30 14:14:40.321313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:40.321299 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e347019e-4fdb-4bd5-a51d-13e08cee3e21" path="/var/lib/kubelet/pods/e347019e-4fdb-4bd5-a51d-13e08cee3e21/volumes" Oct 30 14:14:41.045332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.045296 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp"] Oct 30 14:14:41.049689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.049659 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d2aa13-bmnsp"] Oct 30 14:14:41.084148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.078575 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" event={"ID":"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf","Type":"ContainerStarted","Data":"aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b"} Oct 30 14:14:41.084148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.078672 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" event={"ID":"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf","Type":"ContainerStarted","Data":"6cb89222fb3ef242a327f33933fee444f912bbc138f355f83ca6e113339ec6e6"} Oct 30 14:14:41.091304 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.079261 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:41.093423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.093388 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" event={"ID":"ae95a746-48ba-4686-8f37-ae30206dc275","Type":"ContainerStarted","Data":"0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5"} Oct 30 14:14:41.093594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.093457 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:41.130953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.130899 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" podStartSLOduration=2.130884414 podStartE2EDuration="2.130884414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:41.129148004 +0000 UTC m=+1510.929653471" watchObservedRunningTime="2026-10-30 14:14:41.130884414 +0000 UTC m=+1510.931389875" Oct 30 14:14:41.176509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:41.176446 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" podStartSLOduration=2.176429935 podStartE2EDuration="2.176429935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:41.175621322 +0000 UTC m=+1510.976126766" watchObservedRunningTime="2026-10-30 14:14:41.176429935 +0000 UTC m=+1510.976935380" Oct 30 14:14:42.358256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:42.344702 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8f2355-d43f-4945-986b-0528d04e18cc" path="/var/lib/kubelet/pods/1f8f2355-d43f-4945-986b-0528d04e18cc/volumes" Oct 30 14:14:42.998824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:42.998783 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:14:48.921273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:48.921235 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh"] Oct 30 14:14:48.927486 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:48.927463 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:48.943852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:48.943826 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh"] Oct 30 14:14:48.952283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:48.952224 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-e371c3\"" Oct 30 14:14:48.952443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:48.952363 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-e371c3-dockercfg-qmx6w\"" Oct 30 14:14:49.068212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.064484 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b417b1-8f7a-421b-8db2-4fa9ac539d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh\" (UID: \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:49.068212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.064624 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pccfl\" (UniqueName: \"kubernetes.io/projected/63b417b1-8f7a-421b-8db2-4fa9ac539d94-kube-api-access-pccfl\") pod \"payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh\" (UID: \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:49.145997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.145891 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v"] Oct 30 14:14:49.168395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.167686 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b417b1-8f7a-421b-8db2-4fa9ac539d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh\" (UID: \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:49.168395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.167839 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pccfl\" (UniqueName: \"kubernetes.io/projected/63b417b1-8f7a-421b-8db2-4fa9ac539d94-kube-api-access-pccfl\") pod \"payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh\" (UID: \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:49.169855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.169775 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b417b1-8f7a-421b-8db2-4fa9ac539d94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh\" (UID: \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:49.179675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.179651 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v"] Oct 30 14:14:49.179960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.179892 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:49.195543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.195518 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pccfl\" (UniqueName: \"kubernetes.io/projected/63b417b1-8f7a-421b-8db2-4fa9ac539d94-kube-api-access-pccfl\") pod \"payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh\" (UID: \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:49.243743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.243713 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:49.269940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.269906 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/c1711c3e-b183-4985-8d8a-7a4da293cf1d-kube-api-access-slwl8\") pod \"payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v\" (UID: \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:49.270303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.270275 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1711c3e-b183-4985-8d8a-7a4da293cf1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v\" (UID: \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:49.372635 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.372587 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1711c3e-b183-4985-8d8a-7a4da293cf1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v\" (UID: \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:49.372801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.372701 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/c1711c3e-b183-4985-8d8a-7a4da293cf1d-kube-api-access-slwl8\") pod \"payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v\" (UID: \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:49.373653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.373623 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1711c3e-b183-4985-8d8a-7a4da293cf1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v\" (UID: \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:49.406469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.406433 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/c1711c3e-b183-4985-8d8a-7a4da293cf1d-kube-api-access-slwl8\") pod \"payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v\" (UID: \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:49.497335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.497262 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:49.564739 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:14:49.564706 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b417b1_8f7a_421b_8db2_4fa9ac539d94.slice/crio-45affa56099430479b0d02eaa813c0d9b54e7b34b6846ba53ddd7bab9a005701 WatchSource:0}: Error finding container 45affa56099430479b0d02eaa813c0d9b54e7b34b6846ba53ddd7bab9a005701: Status 404 returned error can't find the container with id 45affa56099430479b0d02eaa813c0d9b54e7b34b6846ba53ddd7bab9a005701 Oct 30 14:14:49.565735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.565708 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh"] Oct 30 14:14:49.840421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:49.840363 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v"] Oct 30 14:14:50.189430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:50.189381 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" event={"ID":"c1711c3e-b183-4985-8d8a-7a4da293cf1d","Type":"ContainerStarted","Data":"134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae"} Oct 30 14:14:50.189903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:50.189449 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" event={"ID":"c1711c3e-b183-4985-8d8a-7a4da293cf1d","Type":"ContainerStarted","Data":"67f2d2257af2127f0b1c10014029954f2edf7ca3cf8189f27c518ee32ec717bd"} Oct 30 14:14:50.189903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:50.189487 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:14:50.191370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:50.191339 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" event={"ID":"63b417b1-8f7a-421b-8db2-4fa9ac539d94","Type":"ContainerStarted","Data":"a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78"} Oct 30 14:14:50.191508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:50.191392 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" event={"ID":"63b417b1-8f7a-421b-8db2-4fa9ac539d94","Type":"ContainerStarted","Data":"45affa56099430479b0d02eaa813c0d9b54e7b34b6846ba53ddd7bab9a005701"} Oct 30 14:14:50.191508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:50.191420 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:14:50.234748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:50.234692 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" podStartSLOduration=2.234675202 podStartE2EDuration="2.234675202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:50.233348199 +0000 UTC m=+1520.033853658" watchObservedRunningTime="2026-10-30 14:14:50.234675202 +0000 UTC m=+1520.035180647" Oct 30 14:14:50.283502 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:50.283447 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" podStartSLOduration=2.2834327930000002 podStartE2EDuration="2.283432793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:50.277958112 +0000 UTC m=+1520.078463555" watchObservedRunningTime="2026-10-30 14:14:50.283432793 +0000 UTC m=+1520.083938236" Oct 30 14:14:52.104275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:52.104248 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:14:53.127333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:53.127307 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:14:55.037543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:55.037507 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw"] Oct 30 14:14:55.050597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:55.050567 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fdb4e5-nmvkw"] Oct 30 14:14:56.318856 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:56.318819 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec71eae-a7c6-4012-bbed-eb9cd934564c" path="/var/lib/kubelet/pods/6ec71eae-a7c6-4012-bbed-eb9cd934564c/volumes" Oct 30 14:14:57.211054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.211013 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx"] Oct 30 14:14:57.215768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.215750 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.236970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.236938 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx"] Oct 30 14:14:57.238177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.238148 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-0efe43-serving-cert\"" Oct 30 14:14:57.307722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.307693 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czbkg\" (UniqueName: \"kubernetes.io/projected/1d8724ac-5136-4a32-b9ab-7f0aa1277938-kube-api-access-czbkg\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.307885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.307731 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.307997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.307965 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8724ac-5136-4a32-b9ab-7f0aa1277938-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.308284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.308256 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.410204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.410162 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.410668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.410301 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czbkg\" (UniqueName: \"kubernetes.io/projected/1d8724ac-5136-4a32-b9ab-7f0aa1277938-kube-api-access-czbkg\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.410668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.410333 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.410668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.410555 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8724ac-5136-4a32-b9ab-7f0aa1277938-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.412582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.411288 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8724ac-5136-4a32-b9ab-7f0aa1277938-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.434512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.415142 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.461635 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.457383 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.465168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.465054 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czbkg\" (UniqueName: \"kubernetes.io/projected/1d8724ac-5136-4a32-b9ab-7f0aa1277938-kube-api-access-czbkg\") pod \"maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.532233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.532178 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:57.853646 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:14:57.853616 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8724ac_5136_4a32_b9ab_7f0aa1277938.slice/crio-5642eab4e5762d8ca8af279750d746596eaa915c404d9a1faf258e261a9a3168 WatchSource:0}: Error finding container 5642eab4e5762d8ca8af279750d746596eaa915c404d9a1faf258e261a9a3168: Status 404 returned error can't find the container with id 5642eab4e5762d8ca8af279750d746596eaa915c404d9a1faf258e261a9a3168 Oct 30 14:14:57.854435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:57.854412 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx"] Oct 30 14:14:58.267248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:58.267177 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" event={"ID":"1d8724ac-5136-4a32-b9ab-7f0aa1277938","Type":"ContainerStarted","Data":"5642eab4e5762d8ca8af279750d746596eaa915c404d9a1faf258e261a9a3168"} Oct 30 14:14:59.276666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:59.276627 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" event={"ID":"1d8724ac-5136-4a32-b9ab-7f0aa1277938","Type":"ContainerStarted","Data":"fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728"} Oct 30 14:14:59.277129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:59.276733 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:14:59.324326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:14:59.324263 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" podStartSLOduration=1.636491011 podStartE2EDuration="2.324244991s" totalImagesPullingTime="687.75398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:59.323775426 +0000 UTC m=+1529.124280872" watchObservedRunningTime="2026-10-30 14:14:59.324244991 +0000 UTC m=+1529.124750435" Oct 30 14:15:00.166153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.166120 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw"] Oct 30 14:15:00.170755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.170726 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8"] Oct 30 14:15:00.170910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.170885 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:00.176183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.176162 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:00.201000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.200955 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8"] Oct 30 14:15:00.204105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.203089 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:15:00.206026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.205980 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn"] Oct 30 14:15:00.211541 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.211520 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:00.219675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.219643 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw"] Oct 30 14:15:00.226825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.226636 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf"] Oct 30 14:15:00.238133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.238111 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn"] Oct 30 14:15:00.238312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.238293 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:00.249074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.249049 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf"] Oct 30 14:15:00.266338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.266312 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27093601-0f6a-4df3-9b48-dd07d68572aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw\" (UID: \"27093601-0f6a-4df3-9b48-dd07d68572aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:00.266439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.266350 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2qw\" (UniqueName: \"kubernetes.io/projected/3072881c-99f8-43ac-b0fe-f57f5a9816b9-kube-api-access-xt2qw\") pod \"maas-api-key-cleanup-29889495-bwnn8\" (UID: \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:00.266554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.266536 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58chr\" (UniqueName: \"kubernetes.io/projected/81cb6b8b-9b98-453b-93af-df69ecfad266-kube-api-access-58chr\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn\" (UID: \"81cb6b8b-9b98-453b-93af-df69ecfad266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:00.266622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.266569 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxf8r\" (UniqueName: \"kubernetes.io/projected/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-kube-api-access-lxf8r\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf\" (UID: \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:00.266788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.266765 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf\" (UID: \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:00.266881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.266856 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3072881c-99f8-43ac-b0fe-f57f5a9816b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889495-bwnn8\" (UID: \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:00.266976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.266958 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgpm\" (UniqueName: \"kubernetes.io/projected/27093601-0f6a-4df3-9b48-dd07d68572aa-kube-api-access-6tgpm\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw\" (UID: \"27093601-0f6a-4df3-9b48-dd07d68572aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:00.267063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.267047 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cb6b8b-9b98-453b-93af-df69ecfad266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn\" (UID: \"81cb6b8b-9b98-453b-93af-df69ecfad266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:00.368431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.368400 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf\" (UID: \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:00.368869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.368450 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3072881c-99f8-43ac-b0fe-f57f5a9816b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889495-bwnn8\" (UID: \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:00.368869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.368520 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgpm\" (UniqueName: \"kubernetes.io/projected/27093601-0f6a-4df3-9b48-dd07d68572aa-kube-api-access-6tgpm\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw\" (UID: \"27093601-0f6a-4df3-9b48-dd07d68572aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:00.368869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.368578 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cb6b8b-9b98-453b-93af-df69ecfad266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn\" (UID: \"81cb6b8b-9b98-453b-93af-df69ecfad266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:00.368869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.368674 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27093601-0f6a-4df3-9b48-dd07d68572aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw\" (UID: \"27093601-0f6a-4df3-9b48-dd07d68572aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:00.368869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.368711 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2qw\" (UniqueName: \"kubernetes.io/projected/3072881c-99f8-43ac-b0fe-f57f5a9816b9-kube-api-access-xt2qw\") pod \"maas-api-key-cleanup-29889495-bwnn8\" (UID: \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:00.369167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.368918 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58chr\" (UniqueName: \"kubernetes.io/projected/81cb6b8b-9b98-453b-93af-df69ecfad266-kube-api-access-58chr\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn\" (UID: \"81cb6b8b-9b98-453b-93af-df69ecfad266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:00.369167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.368972 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxf8r\" (UniqueName: \"kubernetes.io/projected/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-kube-api-access-lxf8r\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf\" (UID: \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:00.369345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.369287 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf\" (UID: \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:00.369345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.369287 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3072881c-99f8-43ac-b0fe-f57f5a9816b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889495-bwnn8\" (UID: \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:00.369459 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.369382 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cb6b8b-9b98-453b-93af-df69ecfad266-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn\" (UID: \"81cb6b8b-9b98-453b-93af-df69ecfad266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:00.369516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.369487 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27093601-0f6a-4df3-9b48-dd07d68572aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw\" (UID: \"27093601-0f6a-4df3-9b48-dd07d68572aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:00.396073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.396046 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgpm\" (UniqueName: \"kubernetes.io/projected/27093601-0f6a-4df3-9b48-dd07d68572aa-kube-api-access-6tgpm\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw\" (UID: \"27093601-0f6a-4df3-9b48-dd07d68572aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:00.398127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.398095 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxf8r\" (UniqueName: \"kubernetes.io/projected/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-kube-api-access-lxf8r\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf\" (UID: \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:00.398381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.398363 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2qw\" (UniqueName: \"kubernetes.io/projected/3072881c-99f8-43ac-b0fe-f57f5a9816b9-kube-api-access-xt2qw\") pod \"maas-api-key-cleanup-29889495-bwnn8\" (UID: \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:00.398452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.398434 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58chr\" (UniqueName: \"kubernetes.io/projected/81cb6b8b-9b98-453b-93af-df69ecfad266-kube-api-access-58chr\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn\" (UID: \"81cb6b8b-9b98-453b-93af-df69ecfad266\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:00.487663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.487642 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:00.498111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.498089 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:00.546388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.546355 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:00.565234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.562519 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:00.800339 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:15:00.800311 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27093601_0f6a_4df3_9b48_dd07d68572aa.slice/crio-72cfe7fd0c68f5d14d042d15fefadfa0e6b57a6f9d7fc341872487974b50df78 WatchSource:0}: Error finding container 72cfe7fd0c68f5d14d042d15fefadfa0e6b57a6f9d7fc341872487974b50df78: Status 404 returned error can't find the container with id 72cfe7fd0c68f5d14d042d15fefadfa0e6b57a6f9d7fc341872487974b50df78 Oct 30 14:15:00.802258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.802230 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw"] Oct 30 14:15:00.818791 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:15:00.818762 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3072881c_99f8_43ac_b0fe_f57f5a9816b9.slice/crio-cd56da5a5a43ad3d80e5565e273cca3794d0ac2cdf9f402263c28a0bfd722e16 WatchSource:0}: Error finding container cd56da5a5a43ad3d80e5565e273cca3794d0ac2cdf9f402263c28a0bfd722e16: Status 404 returned error can't find the container with id cd56da5a5a43ad3d80e5565e273cca3794d0ac2cdf9f402263c28a0bfd722e16 Oct 30 14:15:00.820876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.820846 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8"] Oct 30 14:15:00.918047 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:15:00.918019 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538d60f9_5bcf_48b2_b2ff_7ebd5209bf90.slice/crio-3ab8b8229b33a2927ac444623e8e04429d2f38eedabfa45153a6215e077abc9c WatchSource:0}: Error finding container 3ab8b8229b33a2927ac444623e8e04429d2f38eedabfa45153a6215e077abc9c: Status 404 returned error can't find the container with id 3ab8b8229b33a2927ac444623e8e04429d2f38eedabfa45153a6215e077abc9c Oct 30 14:15:00.921720 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.921668 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf"] Oct 30 14:15:00.927229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:00.927168 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn"] Oct 30 14:15:00.937303 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:15:00.937278 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cb6b8b_9b98_453b_93af_df69ecfad266.slice/crio-4508f4f678e78ffbb1eb7f3ed6fe236fdb8a394a3fd6099b2e55a6b424822728 WatchSource:0}: Error finding container 4508f4f678e78ffbb1eb7f3ed6fe236fdb8a394a3fd6099b2e55a6b424822728: Status 404 returned error can't find the container with id 4508f4f678e78ffbb1eb7f3ed6fe236fdb8a394a3fd6099b2e55a6b424822728 Oct 30 14:15:01.204727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:01.204700 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:15:01.204879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:01.204766 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:15:01.298693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:01.298663 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" event={"ID":"27093601-0f6a-4df3-9b48-dd07d68572aa","Type":"ContainerStarted","Data":"72cfe7fd0c68f5d14d042d15fefadfa0e6b57a6f9d7fc341872487974b50df78"} Oct 30 14:15:01.300315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:01.300283 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" event={"ID":"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90","Type":"ContainerStarted","Data":"3ab8b8229b33a2927ac444623e8e04429d2f38eedabfa45153a6215e077abc9c"} Oct 30 14:15:01.302454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:01.302417 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" event={"ID":"3072881c-99f8-43ac-b0fe-f57f5a9816b9","Type":"ContainerStarted","Data":"8d575ac0a96c3b2df9b9dc1508b12c1aa265bb19679d107756b62ec836e5eed7"} Oct 30 14:15:01.302583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:01.302482 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" event={"ID":"3072881c-99f8-43ac-b0fe-f57f5a9816b9","Type":"ContainerStarted","Data":"cd56da5a5a43ad3d80e5565e273cca3794d0ac2cdf9f402263c28a0bfd722e16"} Oct 30 14:15:01.307483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:01.307420 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" event={"ID":"81cb6b8b-9b98-453b-93af-df69ecfad266","Type":"ContainerStarted","Data":"4508f4f678e78ffbb1eb7f3ed6fe236fdb8a394a3fd6099b2e55a6b424822728"} Oct 30 14:15:01.381081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:01.381031 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" podStartSLOduration=1.381016957 podStartE2EDuration="1.381016957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:01.3765478 +0000 UTC m=+1531.177053245" watchObservedRunningTime="2026-10-30 14:15:01.381016957 +0000 UTC m=+1531.181522457" Oct 30 14:15:02.320582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.320544 5824 generic.go:356] "Generic (PLEG): container finished" podID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerID="8d575ac0a96c3b2df9b9dc1508b12c1aa265bb19679d107756b62ec836e5eed7" exitCode=60 Oct 30 14:15:02.320784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.320651 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" event={"ID":"3072881c-99f8-43ac-b0fe-f57f5a9816b9","Type":"ContainerDied","Data":"8d575ac0a96c3b2df9b9dc1508b12c1aa265bb19679d107756b62ec836e5eed7"} Oct 30 14:15:02.321013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.320996 5824 scope.go:122] "RemoveContainer" containerID="8d575ac0a96c3b2df9b9dc1508b12c1aa265bb19679d107756b62ec836e5eed7" Oct 30 14:15:02.322691 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.322664 5824 generic.go:356] "Generic (PLEG): container finished" podID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerID="b7777cb8283e37b0425e270f635b720ed14bbbfd7e8ea668ed99a632d3be2bc6" exitCode=60 Oct 30 14:15:02.322889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.322746 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" event={"ID":"81cb6b8b-9b98-453b-93af-df69ecfad266","Type":"ContainerDied","Data":"b7777cb8283e37b0425e270f635b720ed14bbbfd7e8ea668ed99a632d3be2bc6"} Oct 30 14:15:02.323020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.322993 5824 scope.go:122] "RemoveContainer" containerID="b7777cb8283e37b0425e270f635b720ed14bbbfd7e8ea668ed99a632d3be2bc6" Oct 30 14:15:02.324960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.324938 5824 generic.go:356] "Generic (PLEG): container finished" podID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerID="87963a77c268aa562ee611d177f39b74065e4c844cc10ee18b8a447f2b8c7c74" exitCode=60 Oct 30 14:15:02.325049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.325025 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" event={"ID":"27093601-0f6a-4df3-9b48-dd07d68572aa","Type":"ContainerDied","Data":"87963a77c268aa562ee611d177f39b74065e4c844cc10ee18b8a447f2b8c7c74"} Oct 30 14:15:02.325312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.325292 5824 scope.go:122] "RemoveContainer" containerID="87963a77c268aa562ee611d177f39b74065e4c844cc10ee18b8a447f2b8c7c74" Oct 30 14:15:02.327176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.327157 5824 generic.go:356] "Generic (PLEG): container finished" podID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerID="7f3e98a1831398e1588a5a34a6f5ae0b55a8a445d8d3585d56ae3e69f910564f" exitCode=7 Oct 30 14:15:02.327276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.327214 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" event={"ID":"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90","Type":"ContainerDied","Data":"7f3e98a1831398e1588a5a34a6f5ae0b55a8a445d8d3585d56ae3e69f910564f"} Oct 30 14:15:02.327512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:02.327494 5824 scope.go:122] "RemoveContainer" containerID="7f3e98a1831398e1588a5a34a6f5ae0b55a8a445d8d3585d56ae3e69f910564f" Oct 30 14:15:03.337807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.337775 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" event={"ID":"27093601-0f6a-4df3-9b48-dd07d68572aa","Type":"ContainerStarted","Data":"5343ea803d60201ba5b8e6b7d3d8508f29bd84943299228fab439d2629e1e211"} Oct 30 14:15:03.339771 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.339745 5824 generic.go:356] "Generic (PLEG): container finished" podID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerID="060669a1339a76d4ce40f5f6b730445c81b6ab3293227292957dd8b7adef713c" exitCode=7 Oct 30 14:15:03.339868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.339829 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" event={"ID":"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90","Type":"ContainerDied","Data":"060669a1339a76d4ce40f5f6b730445c81b6ab3293227292957dd8b7adef713c"} Oct 30 14:15:03.339935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.339887 5824 scope.go:122] "RemoveContainer" containerID="7f3e98a1831398e1588a5a34a6f5ae0b55a8a445d8d3585d56ae3e69f910564f" Oct 30 14:15:03.340150 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.340129 5824 scope.go:122] "RemoveContainer" containerID="060669a1339a76d4ce40f5f6b730445c81b6ab3293227292957dd8b7adef713c" Oct 30 14:15:03.340547 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:03.340523 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf_odh-ai-gateway-infra(538d60f9-5bcf-48b2-b2ff-7ebd5209bf90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" Oct 30 14:15:03.342243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.342220 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" event={"ID":"3072881c-99f8-43ac-b0fe-f57f5a9816b9","Type":"ContainerStarted","Data":"07187abfdff4a9372f06c70f06555abe7cf3f4f0d9aa6898b9ae66e468da43f1"} Oct 30 14:15:03.344517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.344494 5824 generic.go:356] "Generic (PLEG): container finished" podID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerID="15bbdbd6bff70ad41e7bfaaf1f6f899a01f4e03d037815b87129988adff43114" exitCode=60 Oct 30 14:15:03.344593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.344569 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" event={"ID":"81cb6b8b-9b98-453b-93af-df69ecfad266","Type":"ContainerDied","Data":"15bbdbd6bff70ad41e7bfaaf1f6f899a01f4e03d037815b87129988adff43114"} Oct 30 14:15:03.344827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.344813 5824 scope.go:122] "RemoveContainer" containerID="15bbdbd6bff70ad41e7bfaaf1f6f899a01f4e03d037815b87129988adff43114" Oct 30 14:15:03.345128 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:03.345109 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn_odh-ai-gateway-infra(81cb6b8b-9b98-453b-93af-df69ecfad266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" Oct 30 14:15:03.385485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.385442 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" podStartSLOduration=3.385428672 podStartE2EDuration="3.385428672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:03.38234343 +0000 UTC m=+1533.182848875" watchObservedRunningTime="2026-10-30 14:15:03.385428672 +0000 UTC m=+1533.185934148" Oct 30 14:15:03.478381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:03.478342 5824 scope.go:122] "RemoveContainer" containerID="b7777cb8283e37b0425e270f635b720ed14bbbfd7e8ea668ed99a632d3be2bc6" Oct 30 14:15:03.664416 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:03.664369 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:15:04.354489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.354459 5824 scope.go:122] "RemoveContainer" containerID="060669a1339a76d4ce40f5f6b730445c81b6ab3293227292957dd8b7adef713c" Oct 30 14:15:04.354908 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:04.354874 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf_odh-ai-gateway-infra(538d60f9-5bcf-48b2-b2ff-7ebd5209bf90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" Oct 30 14:15:04.356155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.356128 5824 generic.go:356] "Generic (PLEG): container finished" podID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerID="07187abfdff4a9372f06c70f06555abe7cf3f4f0d9aa6898b9ae66e468da43f1" exitCode=60 Oct 30 14:15:04.356295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.356218 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" event={"ID":"3072881c-99f8-43ac-b0fe-f57f5a9816b9","Type":"ContainerDied","Data":"07187abfdff4a9372f06c70f06555abe7cf3f4f0d9aa6898b9ae66e468da43f1"} Oct 30 14:15:04.356295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.356271 5824 scope.go:122] "RemoveContainer" containerID="8d575ac0a96c3b2df9b9dc1508b12c1aa265bb19679d107756b62ec836e5eed7" Oct 30 14:15:04.356459 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.356441 5824 scope.go:122] "RemoveContainer" containerID="07187abfdff4a9372f06c70f06555abe7cf3f4f0d9aa6898b9ae66e468da43f1" Oct 30 14:15:04.356781 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:04.356759 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889495-bwnn8_odh-ai-gateway-infra(3072881c-99f8-43ac-b0fe-f57f5a9816b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" Oct 30 14:15:04.358265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.358246 5824 scope.go:122] "RemoveContainer" containerID="15bbdbd6bff70ad41e7bfaaf1f6f899a01f4e03d037815b87129988adff43114" Oct 30 14:15:04.358565 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:04.358543 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn_odh-ai-gateway-infra(81cb6b8b-9b98-453b-93af-df69ecfad266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" Oct 30 14:15:04.360139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.360120 5824 generic.go:356] "Generic (PLEG): container finished" podID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerID="5343ea803d60201ba5b8e6b7d3d8508f29bd84943299228fab439d2629e1e211" exitCode=60 Oct 30 14:15:04.360224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.360152 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" event={"ID":"27093601-0f6a-4df3-9b48-dd07d68572aa","Type":"ContainerDied","Data":"5343ea803d60201ba5b8e6b7d3d8508f29bd84943299228fab439d2629e1e211"} Oct 30 14:15:04.360447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.360427 5824 scope.go:122] "RemoveContainer" containerID="5343ea803d60201ba5b8e6b7d3d8508f29bd84943299228fab439d2629e1e211" Oct 30 14:15:04.360718 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:04.360698 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw_odh-ai-gateway-infra(27093601-0f6a-4df3-9b48-dd07d68572aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" Oct 30 14:15:04.370582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:04.370567 5824 scope.go:122] "RemoveContainer" containerID="87963a77c268aa562ee611d177f39b74065e4c844cc10ee18b8a447f2b8c7c74" Oct 30 14:15:05.292313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:05.292284 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:15:05.392476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:05.392437 5824 scope.go:122] "RemoveContainer" containerID="5343ea803d60201ba5b8e6b7d3d8508f29bd84943299228fab439d2629e1e211" Oct 30 14:15:05.392911 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:05.392865 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw_odh-ai-gateway-infra(27093601-0f6a-4df3-9b48-dd07d68572aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" Oct 30 14:15:05.397252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:05.395383 5824 scope.go:122] "RemoveContainer" containerID="07187abfdff4a9372f06c70f06555abe7cf3f4f0d9aa6898b9ae66e468da43f1" Oct 30 14:15:05.397252 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:05.395806 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889495-bwnn8_odh-ai-gateway-infra(3072881c-99f8-43ac-b0fe-f57f5a9816b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" Oct 30 14:15:09.069678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:09.069638 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv"] Oct 30 14:15:09.084130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:09.084094 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f02099-ptwrv"] Oct 30 14:15:10.319113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:10.319072 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44e621e-6b30-4220-b7a3-229158196067" path="/var/lib/kubelet/pods/f44e621e-6b30-4220-b7a3-229158196067/volumes" Oct 30 14:15:14.310184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:14.310100 5824 scope.go:122] "RemoveContainer" containerID="15bbdbd6bff70ad41e7bfaaf1f6f899a01f4e03d037815b87129988adff43114" Oct 30 14:15:15.491133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:15.491101 5824 generic.go:356] "Generic (PLEG): container finished" podID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerID="7ed8308fc16a80613b96c4eaca737f321d5e7faa03373c8f7eb2de62487a6758" exitCode=60 Oct 30 14:15:15.491549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:15.491170 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" event={"ID":"81cb6b8b-9b98-453b-93af-df69ecfad266","Type":"ContainerDied","Data":"7ed8308fc16a80613b96c4eaca737f321d5e7faa03373c8f7eb2de62487a6758"} Oct 30 14:15:15.491549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:15.491269 5824 scope.go:122] "RemoveContainer" containerID="15bbdbd6bff70ad41e7bfaaf1f6f899a01f4e03d037815b87129988adff43114" Oct 30 14:15:15.491682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:15.491639 5824 scope.go:122] "RemoveContainer" containerID="7ed8308fc16a80613b96c4eaca737f321d5e7faa03373c8f7eb2de62487a6758" Oct 30 14:15:15.491977 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:15.491945 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn_odh-ai-gateway-infra(81cb6b8b-9b98-453b-93af-df69ecfad266)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" Oct 30 14:15:16.309834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.309805 5824 scope.go:122] "RemoveContainer" containerID="07187abfdff4a9372f06c70f06555abe7cf3f4f0d9aa6898b9ae66e468da43f1" Oct 30 14:15:16.310029 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.310007 5824 scope.go:122] "RemoveContainer" containerID="060669a1339a76d4ce40f5f6b730445c81b6ab3293227292957dd8b7adef713c" Oct 30 14:15:16.553670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.553633 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn"] Oct 30 14:15:16.781676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.781654 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:16.796173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.796148 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cb6b8b-9b98-453b-93af-df69ecfad266-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cb6b8b-9b98-453b-93af-df69ecfad266-maas-api-service-ca\") pod \"81cb6b8b-9b98-453b-93af-df69ecfad266\" (UID: \"81cb6b8b-9b98-453b-93af-df69ecfad266\") " Oct 30 14:15:16.796461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.796438 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cb6b8b-9b98-453b-93af-df69ecfad266-maas-api-service-ca" pod "81cb6b8b-9b98-453b-93af-df69ecfad266" (UID: "81cb6b8b-9b98-453b-93af-df69ecfad266"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:15:16.796856 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.796476 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cb6b8b-9b98-453b-93af-df69ecfad266-kube-api-access-58chr\" (UniqueName: \"kubernetes.io/projected/81cb6b8b-9b98-453b-93af-df69ecfad266-kube-api-access-58chr\") pod \"81cb6b8b-9b98-453b-93af-df69ecfad266\" (UID: \"81cb6b8b-9b98-453b-93af-df69ecfad266\") " Oct 30 14:15:16.796991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.796975 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cb6b8b-9b98-453b-93af-df69ecfad266-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:15:16.798708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.798679 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cb6b8b-9b98-453b-93af-df69ecfad266-kube-api-access-58chr" pod "81cb6b8b-9b98-453b-93af-df69ecfad266" (UID: "81cb6b8b-9b98-453b-93af-df69ecfad266"). InnerVolumeSpecName "kube-api-access-58chr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:16.898722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:16.898642 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58chr\" (UniqueName: \"kubernetes.io/projected/81cb6b8b-9b98-453b-93af-df69ecfad266-kube-api-access-58chr\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:15:17.310022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.309992 5824 scope.go:122] "RemoveContainer" containerID="5343ea803d60201ba5b8e6b7d3d8508f29bd84943299228fab439d2629e1e211" Oct 30 14:15:17.571836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.571748 5824 generic.go:356] "Generic (PLEG): container finished" podID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerID="e58d49604bffe0d840d54159549d6e6ac442ee8599a2726d1b4d1a77be2e41dd" exitCode=60 Oct 30 14:15:17.572599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.572487 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" event={"ID":"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90","Type":"ContainerDied","Data":"e58d49604bffe0d840d54159549d6e6ac442ee8599a2726d1b4d1a77be2e41dd"} Oct 30 14:15:17.572788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.572773 5824 scope.go:122] "RemoveContainer" containerID="060669a1339a76d4ce40f5f6b730445c81b6ab3293227292957dd8b7adef713c" Oct 30 14:15:17.573731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.573707 5824 scope.go:122] "RemoveContainer" containerID="e58d49604bffe0d840d54159549d6e6ac442ee8599a2726d1b4d1a77be2e41dd" Oct 30 14:15:17.574447 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:17.574418 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf_odh-ai-gateway-infra(538d60f9-5bcf-48b2-b2ff-7ebd5209bf90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" Oct 30 14:15:17.581793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.581753 5824 generic.go:356] "Generic (PLEG): container finished" podID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerID="b2d4b61b5856f435056632472d638d447064bb9ed4ca58f758f25394289fa9f6" exitCode=60 Oct 30 14:15:17.582013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.581994 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" event={"ID":"3072881c-99f8-43ac-b0fe-f57f5a9816b9","Type":"ContainerDied","Data":"b2d4b61b5856f435056632472d638d447064bb9ed4ca58f758f25394289fa9f6"} Oct 30 14:15:17.582754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.582731 5824 scope.go:122] "RemoveContainer" containerID="b2d4b61b5856f435056632472d638d447064bb9ed4ca58f758f25394289fa9f6" Oct 30 14:15:17.583441 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:17.583419 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889495-bwnn8_odh-ai-gateway-infra(3072881c-99f8-43ac-b0fe-f57f5a9816b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" Oct 30 14:15:17.585688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.585669 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" event={"ID":"81cb6b8b-9b98-453b-93af-df69ecfad266","Type":"ContainerDied","Data":"4508f4f678e78ffbb1eb7f3ed6fe236fdb8a394a3fd6099b2e55a6b424822728"} Oct 30 14:15:17.585983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.585930 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn" Oct 30 14:15:17.618765 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.618742 5824 scope.go:122] "RemoveContainer" containerID="07187abfdff4a9372f06c70f06555abe7cf3f4f0d9aa6898b9ae66e468da43f1" Oct 30 14:15:17.636909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.636884 5824 scope.go:122] "RemoveContainer" containerID="7ed8308fc16a80613b96c4eaca737f321d5e7faa03373c8f7eb2de62487a6758" Oct 30 14:15:17.719888 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.719857 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn"] Oct 30 14:15:17.725685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:17.725657 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889495-92ngn"] Oct 30 14:15:18.319725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.319699 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" path="/var/lib/kubelet/pods/81cb6b8b-9b98-453b-93af-df69ecfad266/volumes" Oct 30 14:15:18.598303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.598206 5824 generic.go:356] "Generic (PLEG): container finished" podID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerID="ed1ad440107b5bc876ea9d444e58b368264ee78d4d12fd6988fe9a964bc003b4" exitCode=60 Oct 30 14:15:18.598303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.598239 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" event={"ID":"27093601-0f6a-4df3-9b48-dd07d68572aa","Type":"ContainerDied","Data":"ed1ad440107b5bc876ea9d444e58b368264ee78d4d12fd6988fe9a964bc003b4"} Oct 30 14:15:18.598810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.598314 5824 scope.go:122] "RemoveContainer" containerID="5343ea803d60201ba5b8e6b7d3d8508f29bd84943299228fab439d2629e1e211" Oct 30 14:15:18.599596 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.599559 5824 scope.go:122] "RemoveContainer" containerID="ed1ad440107b5bc876ea9d444e58b368264ee78d4d12fd6988fe9a964bc003b4" Oct 30 14:15:18.600226 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:18.600184 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw_odh-ai-gateway-infra(27093601-0f6a-4df3-9b48-dd07d68572aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" Oct 30 14:15:18.643544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.643487 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf"] Oct 30 14:15:18.687175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.687144 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8"] Oct 30 14:15:18.842925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.842891 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:18.869509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.869430 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-kube-api-access-lxf8r\" (UniqueName: \"kubernetes.io/projected/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-kube-api-access-lxf8r\") pod \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\" (UID: \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\") " Oct 30 14:15:18.869944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.869916 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-maas-api-service-ca\") pod \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\" (UID: \"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90\") " Oct 30 14:15:18.870392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.870365 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-maas-api-service-ca" pod "538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" (UID: "538d60f9-5bcf-48b2-b2ff-7ebd5209bf90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:15:18.870683 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.870660 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:15:18.873330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.873306 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-kube-api-access-lxf8r" pod "538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" (UID: "538d60f9-5bcf-48b2-b2ff-7ebd5209bf90"). InnerVolumeSpecName "kube-api-access-lxf8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:18.947933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.947908 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:18.971934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.971912 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3072881c-99f8-43ac-b0fe-f57f5a9816b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3072881c-99f8-43ac-b0fe-f57f5a9816b9-maas-api-service-ca\") pod \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\" (UID: \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\") " Oct 30 14:15:18.972039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.972009 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3072881c-99f8-43ac-b0fe-f57f5a9816b9-kube-api-access-xt2qw\" (UniqueName: \"kubernetes.io/projected/3072881c-99f8-43ac-b0fe-f57f5a9816b9-kube-api-access-xt2qw\") pod \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\" (UID: \"3072881c-99f8-43ac-b0fe-f57f5a9816b9\") " Oct 30 14:15:18.972351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.972321 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3072881c-99f8-43ac-b0fe-f57f5a9816b9-maas-api-service-ca" pod "3072881c-99f8-43ac-b0fe-f57f5a9816b9" (UID: "3072881c-99f8-43ac-b0fe-f57f5a9816b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:15:18.972519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.972498 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxf8r\" (UniqueName: \"kubernetes.io/projected/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90-kube-api-access-lxf8r\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:15:18.972519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.972516 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3072881c-99f8-43ac-b0fe-f57f5a9816b9-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:15:18.974141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:18.974118 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3072881c-99f8-43ac-b0fe-f57f5a9816b9-kube-api-access-xt2qw" pod "3072881c-99f8-43ac-b0fe-f57f5a9816b9" (UID: "3072881c-99f8-43ac-b0fe-f57f5a9816b9"). InnerVolumeSpecName "kube-api-access-xt2qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:19.073958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.073923 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2qw\" (UniqueName: \"kubernetes.io/projected/3072881c-99f8-43ac-b0fe-f57f5a9816b9-kube-api-access-xt2qw\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:15:19.623046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.623003 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" Oct 30 14:15:19.623046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.623023 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf" event={"ID":"538d60f9-5bcf-48b2-b2ff-7ebd5209bf90","Type":"ContainerDied","Data":"3ab8b8229b33a2927ac444623e8e04429d2f38eedabfa45153a6215e077abc9c"} Oct 30 14:15:19.623583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.623103 5824 scope.go:122] "RemoveContainer" containerID="e58d49604bffe0d840d54159549d6e6ac442ee8599a2726d1b4d1a77be2e41dd" Oct 30 14:15:19.624869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.624853 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" Oct 30 14:15:19.624968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.624878 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8" event={"ID":"3072881c-99f8-43ac-b0fe-f57f5a9816b9","Type":"ContainerDied","Data":"cd56da5a5a43ad3d80e5565e273cca3794d0ac2cdf9f402263c28a0bfd722e16"} Oct 30 14:15:19.637930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.636902 5824 scope.go:122] "RemoveContainer" containerID="b2d4b61b5856f435056632472d638d447064bb9ed4ca58f758f25394289fa9f6" Oct 30 14:15:19.666310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.666284 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw"] Oct 30 14:15:19.695042 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.694990 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf"] Oct 30 14:15:19.701603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.701570 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889495-kpwjf"] Oct 30 14:15:19.747237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.744939 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8"] Oct 30 14:15:19.753608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.751982 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889495-bwnn8"] Oct 30 14:15:19.839178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.839153 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:19.887824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.887759 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27093601-0f6a-4df3-9b48-dd07d68572aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27093601-0f6a-4df3-9b48-dd07d68572aa-maas-api-service-ca\") pod \"27093601-0f6a-4df3-9b48-dd07d68572aa\" (UID: \"27093601-0f6a-4df3-9b48-dd07d68572aa\") " Oct 30 14:15:19.888102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.888080 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27093601-0f6a-4df3-9b48-dd07d68572aa-maas-api-service-ca" pod "27093601-0f6a-4df3-9b48-dd07d68572aa" (UID: "27093601-0f6a-4df3-9b48-dd07d68572aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:15:19.888175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.888113 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27093601-0f6a-4df3-9b48-dd07d68572aa-kube-api-access-6tgpm\" (UniqueName: \"kubernetes.io/projected/27093601-0f6a-4df3-9b48-dd07d68572aa-kube-api-access-6tgpm\") pod \"27093601-0f6a-4df3-9b48-dd07d68572aa\" (UID: \"27093601-0f6a-4df3-9b48-dd07d68572aa\") " Oct 30 14:15:19.888768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.888748 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27093601-0f6a-4df3-9b48-dd07d68572aa-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:15:19.890284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.890256 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27093601-0f6a-4df3-9b48-dd07d68572aa-kube-api-access-6tgpm" pod "27093601-0f6a-4df3-9b48-dd07d68572aa" (UID: "27093601-0f6a-4df3-9b48-dd07d68572aa"). InnerVolumeSpecName "kube-api-access-6tgpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:15:19.990437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:19.990398 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tgpm\" (UniqueName: \"kubernetes.io/projected/27093601-0f6a-4df3-9b48-dd07d68572aa-kube-api-access-6tgpm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:15:20.319536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:20.319507 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" path="/var/lib/kubelet/pods/3072881c-99f8-43ac-b0fe-f57f5a9816b9/volumes" Oct 30 14:15:20.319952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:20.319938 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" path="/var/lib/kubelet/pods/538d60f9-5bcf-48b2-b2ff-7ebd5209bf90/volumes" Oct 30 14:15:20.635877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:20.635795 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" Oct 30 14:15:20.635877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:20.635831 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw" event={"ID":"27093601-0f6a-4df3-9b48-dd07d68572aa","Type":"ContainerDied","Data":"72cfe7fd0c68f5d14d042d15fefadfa0e6b57a6f9d7fc341872487974b50df78"} Oct 30 14:15:20.636407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:20.635896 5824 scope.go:122] "RemoveContainer" containerID="ed1ad440107b5bc876ea9d444e58b368264ee78d4d12fd6988fe9a964bc003b4" Oct 30 14:15:20.697290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:20.697256 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw"] Oct 30 14:15:20.710775 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:20.710750 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889495-9f5sw"] Oct 30 14:15:22.318504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:22.318467 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" path="/var/lib/kubelet/pods/27093601-0f6a-4df3-9b48-dd07d68572aa/volumes" Oct 30 14:15:30.756183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:30.756150 5824 scope.go:122] "RemoveContainer" containerID="020db3c1c355b492c31c86dc00f851d8aea64003eb3c44dadd4f9c216fe9853f" Oct 30 14:15:30.767251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:30.767229 5824 scope.go:122] "RemoveContainer" containerID="601127f7bcdc26a32f778c53e8283fa7b814afd1e73c4de03736125c13193d84" Oct 30 14:15:33.643423 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:15:33.643385 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:15:34.170564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.170526 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp"] Oct 30 14:15:34.171287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171271 5824 cpu_manager.go:414] "containerMap: removing container" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.171338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171288 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.171338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171296 5824 cpu_manager.go:414] "containerMap: removing container" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.171338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171302 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.171338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171316 5824 cpu_manager.go:414] "containerMap: removing container" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.171338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171321 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.171338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171330 5824 cpu_manager.go:414] "containerMap: removing container" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.171338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171336 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171354 5824 cpu_manager.go:414] "containerMap: removing container" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171359 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171365 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171369 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171379 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171383 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171398 5824 cpu_manager.go:414] "containerMap: removing container" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171403 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171537 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171545 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171552 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171561 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.171565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171567 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.171946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171575 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.171946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171582 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.171946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171590 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.171946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171598 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.171946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171602 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.171946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.171609 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.177453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.177434 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:34.195244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.195218 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-5704d6-dockercfg-7llh6\"" Oct 30 14:15:34.195475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.195445 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp"] Oct 30 14:15:34.202690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.202650 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-5704d6\"" Oct 30 14:15:34.239409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.237447 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8qpv\" (UniqueName: \"kubernetes.io/projected/f399c067-eb62-47ee-8cfc-733da84d7351-kube-api-access-h8qpv\") pod \"payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp\" (UID: \"f399c067-eb62-47ee-8cfc-733da84d7351\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:34.239409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.237628 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399c067-eb62-47ee-8cfc-733da84d7351-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp\" (UID: \"f399c067-eb62-47ee-8cfc-733da84d7351\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:34.463280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.449452 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8qpv\" (UniqueName: \"kubernetes.io/projected/f399c067-eb62-47ee-8cfc-733da84d7351-kube-api-access-h8qpv\") pod \"payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp\" (UID: \"f399c067-eb62-47ee-8cfc-733da84d7351\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:34.463280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.449931 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399c067-eb62-47ee-8cfc-733da84d7351-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp\" (UID: \"f399c067-eb62-47ee-8cfc-733da84d7351\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:34.463280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.457218 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399c067-eb62-47ee-8cfc-733da84d7351-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp\" (UID: \"f399c067-eb62-47ee-8cfc-733da84d7351\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:34.501401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.501366 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8qpv\" (UniqueName: \"kubernetes.io/projected/f399c067-eb62-47ee-8cfc-733da84d7351-kube-api-access-h8qpv\") pod \"payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp\" (UID: \"f399c067-eb62-47ee-8cfc-733da84d7351\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:34.506449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.506410 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc"] Oct 30 14:15:34.509337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.509264 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.509514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.509498 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3072881c-99f8-43ac-b0fe-f57f5a9816b9" containerName="cleanup" Oct 30 14:15:34.509707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.509695 5824 cpu_manager.go:414] "containerMap: removing container" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.509783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.509774 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.509870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.509862 5824 cpu_manager.go:414] "containerMap: removing container" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.509933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.509925 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538d60f9-5bcf-48b2-b2ff-7ebd5209bf90" containerName="cleanup" Oct 30 14:15:34.510005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.509997 5824 cpu_manager.go:414] "containerMap: removing container" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.510066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.510058 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27093601-0f6a-4df3-9b48-dd07d68572aa" containerName="cleanup" Oct 30 14:15:34.510328 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.510312 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cb6b8b-9b98-453b-93af-df69ecfad266" containerName="cleanup" Oct 30 14:15:34.514789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.514763 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc"] Oct 30 14:15:34.514910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.514894 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:34.676342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.676306 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dl95\" (UniqueName: \"kubernetes.io/projected/eaa3f864-0b20-41ca-9594-61803ab6411f-kube-api-access-4dl95\") pod \"payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc\" (UID: \"eaa3f864-0b20-41ca-9594-61803ab6411f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:34.676711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.676441 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa3f864-0b20-41ca-9594-61803ab6411f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc\" (UID: \"eaa3f864-0b20-41ca-9594-61803ab6411f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:34.778055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.777966 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dl95\" (UniqueName: \"kubernetes.io/projected/eaa3f864-0b20-41ca-9594-61803ab6411f-kube-api-access-4dl95\") pod \"payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc\" (UID: \"eaa3f864-0b20-41ca-9594-61803ab6411f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:34.778234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.778094 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa3f864-0b20-41ca-9594-61803ab6411f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc\" (UID: \"eaa3f864-0b20-41ca-9594-61803ab6411f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:34.778896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.778875 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa3f864-0b20-41ca-9594-61803ab6411f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc\" (UID: \"eaa3f864-0b20-41ca-9594-61803ab6411f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:34.792565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.792544 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:34.804350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.804328 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dl95\" (UniqueName: \"kubernetes.io/projected/eaa3f864-0b20-41ca-9594-61803ab6411f-kube-api-access-4dl95\") pod \"payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc\" (UID: \"eaa3f864-0b20-41ca-9594-61803ab6411f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:34.838929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:34.838894 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:35.166550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.164242 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp"] Oct 30 14:15:35.401456 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:15:35.401417 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa3f864_0b20_41ca_9594_61803ab6411f.slice/crio-b62ec6a15450ebad16f0ebff05fec38bedff8537c57cd1175a9b25b75cfa3484 WatchSource:0}: Error finding container b62ec6a15450ebad16f0ebff05fec38bedff8537c57cd1175a9b25b75cfa3484: Status 404 returned error can't find the container with id b62ec6a15450ebad16f0ebff05fec38bedff8537c57cd1175a9b25b75cfa3484 Oct 30 14:15:35.403161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.403138 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc"] Oct 30 14:15:35.772523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.772417 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" event={"ID":"eaa3f864-0b20-41ca-9594-61803ab6411f","Type":"ContainerStarted","Data":"5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a"} Oct 30 14:15:35.772523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.772488 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" event={"ID":"eaa3f864-0b20-41ca-9594-61803ab6411f","Type":"ContainerStarted","Data":"b62ec6a15450ebad16f0ebff05fec38bedff8537c57cd1175a9b25b75cfa3484"} Oct 30 14:15:35.772523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.772521 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:35.774354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.774326 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" event={"ID":"f399c067-eb62-47ee-8cfc-733da84d7351","Type":"ContainerStarted","Data":"f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26"} Oct 30 14:15:35.774472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.774373 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" event={"ID":"f399c067-eb62-47ee-8cfc-733da84d7351","Type":"ContainerStarted","Data":"f151a7bd8495cb0825edc099017500014f3c29d12041ed5aa94a615e441b4e5a"} Oct 30 14:15:35.774472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.774463 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:35.820591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.820548 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" podStartSLOduration=1.820535035 podStartE2EDuration="1.820535035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:35.817867301 +0000 UTC m=+1565.618372758" watchObservedRunningTime="2026-10-30 14:15:35.820535035 +0000 UTC m=+1565.621040522" Oct 30 14:15:35.863966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:35.863910 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" podStartSLOduration=1.863898528 podStartE2EDuration="1.863898528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:35.862541428 +0000 UTC m=+1565.663046871" watchObservedRunningTime="2026-10-30 14:15:35.863898528 +0000 UTC m=+1565.664403972" Oct 30 14:15:44.036581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:44.036545 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm"] Oct 30 14:15:44.049737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:44.049706 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bc95d2e-bnvfm"] Oct 30 14:15:44.321579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:44.321505 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6331517a-78ea-444f-a92c-219a726b9bbb" path="/var/lib/kubelet/pods/6331517a-78ea-444f-a92c-219a726b9bbb/volumes" Oct 30 14:15:45.349282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.349239 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs"] Oct 30 14:15:45.355204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.355161 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn"] Oct 30 14:15:45.355332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.355273 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:45.359909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.359887 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:45.375336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.375284 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs"] Oct 30 14:15:45.381206 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.381167 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn"] Oct 30 14:15:45.386770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.386494 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a59ecb\"" Oct 30 14:15:45.386770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.386562 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a59ecb-dockercfg-fbprr\"" Oct 30 14:15:45.527214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.522421 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn\" (UID: \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:45.527214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.522502 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m96hj\" (UniqueName: \"kubernetes.io/projected/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-kube-api-access-m96hj\") pod \"payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn\" (UID: \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:45.527214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.522994 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8lh\" (UniqueName: \"kubernetes.io/projected/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-kube-api-access-4j8lh\") pod \"payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs\" (UID: \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:45.527214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.523095 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs\" (UID: \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:45.625368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.625286 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8lh\" (UniqueName: \"kubernetes.io/projected/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-kube-api-access-4j8lh\") pod \"payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs\" (UID: \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:45.625368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.625356 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs\" (UID: \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:45.625603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.625405 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn\" (UID: \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:45.625603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.625454 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m96hj\" (UniqueName: \"kubernetes.io/projected/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-kube-api-access-m96hj\") pod \"payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn\" (UID: \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:45.626098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.626071 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs\" (UID: \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:45.626226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.626170 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn\" (UID: \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:45.652930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.652870 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8lh\" (UniqueName: \"kubernetes.io/projected/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-kube-api-access-4j8lh\") pod \"payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs\" (UID: \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:45.652930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.652918 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m96hj\" (UniqueName: \"kubernetes.io/projected/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-kube-api-access-m96hj\") pod \"payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn\" (UID: \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:45.684281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.684244 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:45.695748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:45.695710 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:46.029492 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:15:46.029456 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1277a3_8129_4d26_9e3e_612d4f7cb14a.slice/crio-38d2f79151ee87421d258e3c148e9187bf60eb4134d18f0bced5d98b213627f5 WatchSource:0}: Error finding container 38d2f79151ee87421d258e3c148e9187bf60eb4134d18f0bced5d98b213627f5: Status 404 returned error can't find the container with id 38d2f79151ee87421d258e3c148e9187bf60eb4134d18f0bced5d98b213627f5 Oct 30 14:15:46.031607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.031579 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs"] Oct 30 14:15:46.055688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.055651 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn"] Oct 30 14:15:46.784321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.784283 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:15:46.785008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.784983 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:15:46.880484 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.880445 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" event={"ID":"99d95ba4-4fb9-46d2-8b43-08f95c4241bb","Type":"ContainerStarted","Data":"4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94"} Oct 30 14:15:46.880662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.880502 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" event={"ID":"99d95ba4-4fb9-46d2-8b43-08f95c4241bb","Type":"ContainerStarted","Data":"78419e7640c9f3a373b43e2192e8b277b4ceed7e6145a6e8b650e7d86e1bc664"} Oct 30 14:15:46.880662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.880531 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:15:46.886259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.885491 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" event={"ID":"7e1277a3-8129-4d26-9e3e-612d4f7cb14a","Type":"ContainerStarted","Data":"660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f"} Oct 30 14:15:46.886259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.885544 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" event={"ID":"7e1277a3-8129-4d26-9e3e-612d4f7cb14a","Type":"ContainerStarted","Data":"38d2f79151ee87421d258e3c148e9187bf60eb4134d18f0bced5d98b213627f5"} Oct 30 14:15:46.886259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.886238 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:46.937354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.937299 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" podStartSLOduration=1.9372810839999999 podStartE2EDuration="1.937281084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:46.931630121 +0000 UTC m=+1576.732135566" watchObservedRunningTime="2026-10-30 14:15:46.937281084 +0000 UTC m=+1576.737786526" Oct 30 14:15:46.978096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:46.978039 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" podStartSLOduration=1.9780247800000001 podStartE2EDuration="1.97802478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:46.976055911 +0000 UTC m=+1576.776561356" watchObservedRunningTime="2026-10-30 14:15:46.97802478 +0000 UTC m=+1576.778530251" Oct 30 14:15:55.484531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.484495 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48"] Oct 30 14:15:55.489577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.489556 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.508846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.508820 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48"] Oct 30 14:15:55.530636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.530610 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-5704d6-serving-cert\"" Oct 30 14:15:55.621414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.621381 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdff\" (UniqueName: \"kubernetes.io/projected/5932998f-68b0-4495-8616-e45762f19574-kube-api-access-kpdff\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.621614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.621421 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.621764 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.621741 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.621902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.621885 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932998f-68b0-4495-8616-e45762f19574-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.723831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.723793 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932998f-68b0-4495-8616-e45762f19574-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.724026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.723862 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdff\" (UniqueName: \"kubernetes.io/projected/5932998f-68b0-4495-8616-e45762f19574-kube-api-access-kpdff\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.724026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.723903 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.724233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.724181 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.724618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.724594 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932998f-68b0-4495-8616-e45762f19574-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.726424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.726401 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.726424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.726419 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.755309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.755133 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdff\" (UniqueName: \"kubernetes.io/projected/5932998f-68b0-4495-8616-e45762f19574-kube-api-access-kpdff\") pod \"maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:55.806167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:55.806133 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:56.132056 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:15:56.132027 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5932998f_68b0_4495_8616_e45762f19574.slice/crio-37f276128561a3d15f2b26e23b51bc987261349506bf9da2ff7e3a665a216d95 WatchSource:0}: Error finding container 37f276128561a3d15f2b26e23b51bc987261349506bf9da2ff7e3a665a216d95: Status 404 returned error can't find the container with id 37f276128561a3d15f2b26e23b51bc987261349506bf9da2ff7e3a665a216d95 Oct 30 14:15:56.133650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:56.133630 5824 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:15:56.134501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:56.134479 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48"] Oct 30 14:15:56.994599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:56.994558 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" event={"ID":"5932998f-68b0-4495-8616-e45762f19574","Type":"ContainerStarted","Data":"17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34"} Oct 30 14:15:56.995227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:56.994624 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" event={"ID":"5932998f-68b0-4495-8616-e45762f19574","Type":"ContainerStarted","Data":"37f276128561a3d15f2b26e23b51bc987261349506bf9da2ff7e3a665a216d95"} Oct 30 14:15:56.995227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:56.994919 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:15:57.044861 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:57.044801 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" podStartSLOduration=1.639331954 podStartE2EDuration="2.044784513s" totalImagesPullingTime="405.452559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:15:57.042808385 +0000 UTC m=+1586.843313830" watchObservedRunningTime="2026-10-30 14:15:57.044784513 +0000 UTC m=+1586.845289957" Oct 30 14:15:57.904379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:57.903869 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:15:57.906230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:15:57.905678 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:16:02.211659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.211617 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8"] Oct 30 14:16:02.216988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.216964 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:02.237726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.237697 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8"] Oct 30 14:16:02.246591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.246564 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-01955278\"" Oct 30 14:16:02.247751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.247719 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6507c54e-2da9-4fb6-9014-14da28aa9ec8-plugins-config-volume\") pod \"payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8\" (UID: \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\") " pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:02.248029 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.247996 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtvn\" (UniqueName: \"kubernetes.io/projected/6507c54e-2da9-4fb6-9014-14da28aa9ec8-kube-api-access-4vtvn\") pod \"payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8\" (UID: \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\") " pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:02.328933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.328896 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq"] Oct 30 14:16:02.333997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.333972 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:02.374215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.367097 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6507c54e-2da9-4fb6-9014-14da28aa9ec8-plugins-config-volume\") pod \"payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8\" (UID: \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\") " pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:02.374215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.368085 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6507c54e-2da9-4fb6-9014-14da28aa9ec8-plugins-config-volume\") pod \"payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8\" (UID: \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\") " pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:02.374215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.368362 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtvn\" (UniqueName: \"kubernetes.io/projected/6507c54e-2da9-4fb6-9014-14da28aa9ec8-kube-api-access-4vtvn\") pod \"payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8\" (UID: \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\") " pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:02.374215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.371689 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-01955278-dockercfg-l9qt5\"" Oct 30 14:16:02.379469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.378939 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq"] Oct 30 14:16:02.484318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.435961 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtvn\" (UniqueName: \"kubernetes.io/projected/6507c54e-2da9-4fb6-9014-14da28aa9ec8-kube-api-access-4vtvn\") pod \"payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8\" (UID: \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\") " pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:02.484318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.470072 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvtm\" (UniqueName: \"kubernetes.io/projected/adec2d68-9246-43fe-9c98-22fbb8891fb6-kube-api-access-wmvtm\") pod \"payload-pre-processing-e2e-mt-01955278-f55689567-97lbq\" (UID: \"adec2d68-9246-43fe-9c98-22fbb8891fb6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:02.484318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.470274 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adec2d68-9246-43fe-9c98-22fbb8891fb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-01955278-f55689567-97lbq\" (UID: \"adec2d68-9246-43fe-9c98-22fbb8891fb6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:02.534263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.534223 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:02.573144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.573103 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adec2d68-9246-43fe-9c98-22fbb8891fb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-01955278-f55689567-97lbq\" (UID: \"adec2d68-9246-43fe-9c98-22fbb8891fb6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:02.573416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.573397 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvtm\" (UniqueName: \"kubernetes.io/projected/adec2d68-9246-43fe-9c98-22fbb8891fb6-kube-api-access-wmvtm\") pod \"payload-pre-processing-e2e-mt-01955278-f55689567-97lbq\" (UID: \"adec2d68-9246-43fe-9c98-22fbb8891fb6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:02.574751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.574724 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adec2d68-9246-43fe-9c98-22fbb8891fb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-01955278-f55689567-97lbq\" (UID: \"adec2d68-9246-43fe-9c98-22fbb8891fb6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:02.613753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.613718 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvtm\" (UniqueName: \"kubernetes.io/projected/adec2d68-9246-43fe-9c98-22fbb8891fb6-kube-api-access-wmvtm\") pod \"payload-pre-processing-e2e-mt-01955278-f55689567-97lbq\" (UID: \"adec2d68-9246-43fe-9c98-22fbb8891fb6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:02.686965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.685953 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:02.893587 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:16:02.893542 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6507c54e_2da9_4fb6_9014_14da28aa9ec8.slice/crio-8c6fd1a59528b957266c5f587d77e76c3391239db21525a783019a2c0e740cb4 WatchSource:0}: Error finding container 8c6fd1a59528b957266c5f587d77e76c3391239db21525a783019a2c0e740cb4: Status 404 returned error can't find the container with id 8c6fd1a59528b957266c5f587d77e76c3391239db21525a783019a2c0e740cb4 Oct 30 14:16:02.895289 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:02.895245 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8"] Oct 30 14:16:03.013045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:03.013018 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:16:03.029341 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:16:03.029260 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadec2d68_9246_43fe_9c98_22fbb8891fb6.slice/crio-2d0ebe7d30af9221da58c09fa824dac60d3431ba7ba6b5ad84dca3fabfaeec6a WatchSource:0}: Error finding container 2d0ebe7d30af9221da58c09fa824dac60d3431ba7ba6b5ad84dca3fabfaeec6a: Status 404 returned error can't find the container with id 2d0ebe7d30af9221da58c09fa824dac60d3431ba7ba6b5ad84dca3fabfaeec6a Oct 30 14:16:03.034754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:03.034685 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq"] Oct 30 14:16:03.053056 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:03.053019 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" event={"ID":"adec2d68-9246-43fe-9c98-22fbb8891fb6","Type":"ContainerStarted","Data":"2d0ebe7d30af9221da58c09fa824dac60d3431ba7ba6b5ad84dca3fabfaeec6a"} Oct 30 14:16:03.055180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:03.055151 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" event={"ID":"6507c54e-2da9-4fb6-9014-14da28aa9ec8","Type":"ContainerStarted","Data":"46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364"} Oct 30 14:16:03.055355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:03.055255 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:03.055355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:03.055286 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" event={"ID":"6507c54e-2da9-4fb6-9014-14da28aa9ec8","Type":"ContainerStarted","Data":"8c6fd1a59528b957266c5f587d77e76c3391239db21525a783019a2c0e740cb4"} Oct 30 14:16:03.119244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:03.119162 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" podStartSLOduration=1.119142331 podStartE2EDuration="1.119142331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:03.112508153 +0000 UTC m=+1592.913013598" watchObservedRunningTime="2026-10-30 14:16:03.119142331 +0000 UTC m=+1592.919647775" Oct 30 14:16:03.644215 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:16:03.643617 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:16:04.101215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:04.082873 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" event={"ID":"adec2d68-9246-43fe-9c98-22fbb8891fb6","Type":"ContainerStarted","Data":"92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997"} Oct 30 14:16:04.101215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:04.082940 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:04.179215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:04.177720 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" podStartSLOduration=2.177697629 podStartE2EDuration="2.177697629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:04.130653718 +0000 UTC m=+1593.931159163" watchObservedRunningTime="2026-10-30 14:16:04.177697629 +0000 UTC m=+1593.978203074" Oct 30 14:16:09.787465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:09.787432 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972"] Oct 30 14:16:09.793743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:09.793706 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:09.820354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:09.820320 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-a59ecb-serving-cert\"" Oct 30 14:16:09.822454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:09.822423 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972"] Oct 30 14:16:09.923166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:09.923130 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:09.923620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:09.923593 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276ef28-e265-4ee5-9e4b-da715f993aaa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:09.923716 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:09.923695 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:09.923766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:09.923738 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvqw4\" (UniqueName: \"kubernetes.io/projected/4276ef28-e265-4ee5-9e4b-da715f993aaa-kube-api-access-qvqw4\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.027101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.027067 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvqw4\" (UniqueName: \"kubernetes.io/projected/4276ef28-e265-4ee5-9e4b-da715f993aaa-kube-api-access-qvqw4\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.027325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.027144 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.027439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.027423 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276ef28-e265-4ee5-9e4b-da715f993aaa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.027501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.027477 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.028492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.028467 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276ef28-e265-4ee5-9e4b-da715f993aaa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.030486 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.030463 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.030824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.030799 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.054062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.053992 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvqw4\" (UniqueName: \"kubernetes.io/projected/4276ef28-e265-4ee5-9e4b-da715f993aaa-kube-api-access-qvqw4\") pod \"maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.122323 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.122289 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:10.473591 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:16:10.473564 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4276ef28_e265_4ee5_9e4b_da715f993aaa.slice/crio-3c65df5c0ca64a4102a25e4f15a73e0bd4ee79f8b2b84ca7c2880a96e3e8dd01 WatchSource:0}: Error finding container 3c65df5c0ca64a4102a25e4f15a73e0bd4ee79f8b2b84ca7c2880a96e3e8dd01: Status 404 returned error can't find the container with id 3c65df5c0ca64a4102a25e4f15a73e0bd4ee79f8b2b84ca7c2880a96e3e8dd01 Oct 30 14:16:10.475141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:10.475115 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972"] Oct 30 14:16:11.157685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:11.157574 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" event={"ID":"4276ef28-e265-4ee5-9e4b-da715f993aaa","Type":"ContainerStarted","Data":"ec2120ab1545f223a4ed83e9ded93746b8826e32c8cf03b322896f0776b60ea7"} Oct 30 14:16:11.157685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:11.157650 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" event={"ID":"4276ef28-e265-4ee5-9e4b-da715f993aaa","Type":"ContainerStarted","Data":"3c65df5c0ca64a4102a25e4f15a73e0bd4ee79f8b2b84ca7c2880a96e3e8dd01"} Oct 30 14:16:11.158790 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:11.158767 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:11.206874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:11.206809 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" podStartSLOduration=1.796431869 podStartE2EDuration="2.206788896s" totalImagesPullingTime="410.357027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:11.205048675 +0000 UTC m=+1601.005554153" watchObservedRunningTime="2026-10-30 14:16:11.206788896 +0000 UTC m=+1601.007294342" Oct 30 14:16:14.086636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:14.086601 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:16:15.095553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:15.095526 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:16:16.043960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:16.043921 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs"] Oct 30 14:16:16.058227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:16.058176 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7c83787-7fncs"] Oct 30 14:16:16.319058 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:16.318979 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427455e4-4f52-48a8-a83b-769a7fd6ee86" path="/var/lib/kubelet/pods/427455e4-4f52-48a8-a83b-769a7fd6ee86/volumes" Oct 30 14:16:18.179662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:18.179634 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:16:25.073211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:25.062243 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv"] Oct 30 14:16:25.136738 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:25.136674 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02dd168a-hvpvv"] Oct 30 14:16:26.433311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:26.433273 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70eaab3-231a-406f-850e-a8184dd7a0d4" path="/var/lib/kubelet/pods/a70eaab3-231a-406f-850e-a8184dd7a0d4/volumes" Oct 30 14:16:30.843480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:30.843446 5824 scope.go:122] "RemoveContainer" containerID="1582cbcd5cffad297d0dd940bd205ebef557a0b2515fb0eb1b5ff1fa3e5999b9" Oct 30 14:16:30.854119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:30.854099 5824 scope.go:122] "RemoveContainer" containerID="54f1d15d5b958873cd3099117da40a88af3d92913d06601efe9cb7445b648884" Oct 30 14:16:30.864205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:30.864098 5824 scope.go:122] "RemoveContainer" containerID="87c842d0d9c4d66624bcc9f4b75bb76f5f2cf7daa80ac68733d42ea931ef2c20" Oct 30 14:16:30.878083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:30.878055 5824 scope.go:122] "RemoveContainer" containerID="3a1f54c266d3b3f70ef2bbc2062b8e1346db87425e314582e31e0d9d7462a42a" Oct 30 14:16:33.649613 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:16:33.649466 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:16:42.041976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:42.041900 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db"] Oct 30 14:16:42.054572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:42.054547 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76275419-765db"] Oct 30 14:16:42.318839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:42.318769 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19ca504-ed6b-4c30-a3e6-fb77c38b0433" path="/var/lib/kubelet/pods/d19ca504-ed6b-4c30-a3e6-fb77c38b0433/volumes" Oct 30 14:16:45.686848 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.686811 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw"] Oct 30 14:16:45.693838 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.693812 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:45.711266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.711239 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw"] Oct 30 14:16:45.719503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.719466 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-228454\"" Oct 30 14:16:45.719503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.719494 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-228454-dockercfg-r4dl7\"" Oct 30 14:16:45.730743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.730716 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7"] Oct 30 14:16:45.743048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.736810 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:45.748659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.748630 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7"] Oct 30 14:16:45.789230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.789176 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g598\" (UniqueName: \"kubernetes.io/projected/5fe90a87-f432-4e8e-8564-45e956c10af7-kube-api-access-6g598\") pod \"payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw\" (UID: \"5fe90a87-f432-4e8e-8564-45e956c10af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:45.789572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.789548 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9mg\" (UniqueName: \"kubernetes.io/projected/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-kube-api-access-jr9mg\") pod \"payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7\" (UID: \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:45.789761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.789739 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe90a87-f432-4e8e-8564-45e956c10af7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw\" (UID: \"5fe90a87-f432-4e8e-8564-45e956c10af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:45.790134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.790114 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7\" (UID: \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:45.902831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.898361 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g598\" (UniqueName: \"kubernetes.io/projected/5fe90a87-f432-4e8e-8564-45e956c10af7-kube-api-access-6g598\") pod \"payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw\" (UID: \"5fe90a87-f432-4e8e-8564-45e956c10af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:45.908122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.908087 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9mg\" (UniqueName: \"kubernetes.io/projected/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-kube-api-access-jr9mg\") pod \"payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7\" (UID: \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:45.908414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.908395 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe90a87-f432-4e8e-8564-45e956c10af7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw\" (UID: \"5fe90a87-f432-4e8e-8564-45e956c10af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:45.908744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.908725 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7\" (UID: \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:45.910184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.909816 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe90a87-f432-4e8e-8564-45e956c10af7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw\" (UID: \"5fe90a87-f432-4e8e-8564-45e956c10af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:45.910567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.910536 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7\" (UID: \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:45.935775 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.935720 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g598\" (UniqueName: \"kubernetes.io/projected/5fe90a87-f432-4e8e-8564-45e956c10af7-kube-api-access-6g598\") pod \"payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw\" (UID: \"5fe90a87-f432-4e8e-8564-45e956c10af7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:45.937117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:45.937046 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9mg\" (UniqueName: \"kubernetes.io/projected/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-kube-api-access-jr9mg\") pod \"payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7\" (UID: \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:46.010379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.010342 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:46.061032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.060733 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:46.384957 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:16:46.384928 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe90a87_f432_4e8e_8564_45e956c10af7.slice/crio-cd663dccf4a6532b71b2a08cb15da337d71163678e0ab45ee79bd070a94211df WatchSource:0}: Error finding container cd663dccf4a6532b71b2a08cb15da337d71163678e0ab45ee79bd070a94211df: Status 404 returned error can't find the container with id cd663dccf4a6532b71b2a08cb15da337d71163678e0ab45ee79bd070a94211df Oct 30 14:16:46.387149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.387128 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw"] Oct 30 14:16:46.431127 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:16:46.431089 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04ffdb6_b2dc_4cd6_95b3_d860e3f648f7.slice/crio-e198d4123bb7ce1d54f43ce0b7f342ada736b50aa0d97312093a34f5e9766691 WatchSource:0}: Error finding container e198d4123bb7ce1d54f43ce0b7f342ada736b50aa0d97312093a34f5e9766691: Status 404 returned error can't find the container with id e198d4123bb7ce1d54f43ce0b7f342ada736b50aa0d97312093a34f5e9766691 Oct 30 14:16:46.434374 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.434344 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7"] Oct 30 14:16:46.628547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.628511 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" event={"ID":"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7","Type":"ContainerStarted","Data":"a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb"} Oct 30 14:16:46.628741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.628570 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" event={"ID":"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7","Type":"ContainerStarted","Data":"e198d4123bb7ce1d54f43ce0b7f342ada736b50aa0d97312093a34f5e9766691"} Oct 30 14:16:46.628741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.628596 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:46.630662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.630635 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" event={"ID":"5fe90a87-f432-4e8e-8564-45e956c10af7","Type":"ContainerStarted","Data":"d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84"} Oct 30 14:16:46.630982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.630683 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" event={"ID":"5fe90a87-f432-4e8e-8564-45e956c10af7","Type":"ContainerStarted","Data":"cd663dccf4a6532b71b2a08cb15da337d71163678e0ab45ee79bd070a94211df"} Oct 30 14:16:46.630982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.630727 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:16:46.676905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.676857 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" podStartSLOduration=1.676843371 podStartE2EDuration="1.676843371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:46.674448702 +0000 UTC m=+1636.474954145" watchObservedRunningTime="2026-10-30 14:16:46.676843371 +0000 UTC m=+1636.477348815" Oct 30 14:16:46.719450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:46.719358 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" podStartSLOduration=1.7193415459999999 podStartE2EDuration="1.719341546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:16:46.718817964 +0000 UTC m=+1636.519323407" watchObservedRunningTime="2026-10-30 14:16:46.719341546 +0000 UTC m=+1636.519846990" Oct 30 14:16:53.070216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:53.069279 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv"] Oct 30 14:16:53.083805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:53.083773 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7232e9b6-dxdfv"] Oct 30 14:16:54.376614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:54.376563 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e79b06-f257-4c49-b57c-315d27cb7548" path="/var/lib/kubelet/pods/66e79b06-f257-4c49-b57c-315d27cb7548/volumes" Oct 30 14:16:56.493228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:56.490356 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd"] Oct 30 14:16:56.493228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:56.490614 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" containerID="cri-o://89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9" gracePeriod=30 Oct 30 14:16:57.224314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:57.224282 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:16:57.224500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:57.224338 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:16:57.642843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:57.642746 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:16:57.644117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:16:57.644095 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:17:01.740015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.739990 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:17:01.776278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776185 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-certs\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776276 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-credential-socket\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776321 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-envoy\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776350 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istiod-ca-cert\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776432 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-token\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-token\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776434 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-certs" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:17:01.776676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776491 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-kube-api-access-29jkv\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-kube-api-access-29jkv\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776586 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-data\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776602 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-credential-socket" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:17:01.776897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776694 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-podinfo\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776784 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-socket\") pod \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\" (UID: \"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d\") " Oct 30 14:17:01.776897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776844 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-data" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:17:01.777040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.776762 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istiod-ca-cert" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:17:01.777159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.777099 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-socket" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:17:01.777656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.777637 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.777741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.777659 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.777741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.777673 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.777741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.777686 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.777741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.777698 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.778741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.778719 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-token" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:17:01.778893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.778863 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-kube-api-access-29jkv" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "kube-api-access-29jkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:17:01.778985 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.778922 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-envoy" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:17:01.778985 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.778942 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-podinfo" pod "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" (UID: "d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:17:01.879545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.879501 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.879545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.879537 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.879545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.879551 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29jkv\" (UniqueName: \"kubernetes.io/projected/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-kube-api-access-29jkv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.879849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.879562 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:01.951676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.951636 5824 generic.go:356] "Generic (PLEG): container finished" podID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerID="89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9" exitCode=0 Oct 30 14:17:01.951839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.951690 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" Oct 30 14:17:01.951839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.951708 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" event={"ID":"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d","Type":"ContainerDied","Data":"89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9"} Oct 30 14:17:01.951839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.951786 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd" event={"ID":"d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d","Type":"ContainerDied","Data":"31440b566443f7a0af599d7e28b84f71cbcdc81f1f37cb16bfc5fd2566e3213d"} Oct 30 14:17:01.951839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.951811 5824 scope.go:122] "RemoveContainer" containerID="89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9" Oct 30 14:17:01.964711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.964689 5824 scope.go:122] "RemoveContainer" containerID="89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9" Oct 30 14:17:01.965036 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:17:01.965017 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9\": container with ID starting with 89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9 not found: ID does not exist" containerID="89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9" Oct 30 14:17:01.965156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:01.965048 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9"} err="failed to get container status \"89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9\": rpc error: code = NotFound desc = could not find container \"89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9\": container with ID starting with 89ab363137a117f618f1a0d9742f4f2a76403f282874c30f822a8d1baa6bd8c9 not found: ID does not exist" Oct 30 14:17:02.013860 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:02.013830 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd"] Oct 30 14:17:02.017623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:02.017600 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-2c9jd"] Oct 30 14:17:02.320475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:02.320439 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" path="/var/lib/kubelet/pods/d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d/volumes" Oct 30 14:17:03.642781 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:17:03.642741 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:17:05.038512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:05.038480 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7"] Oct 30 14:17:05.056575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:05.056536 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd80aa4b-cqxn7"] Oct 30 14:17:06.320565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:06.320534 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159bbda6-cd39-4582-9217-b3b2c7c11a93" path="/var/lib/kubelet/pods/159bbda6-cd39-4582-9217-b3b2c7c11a93/volumes" Oct 30 14:17:09.047953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:09.047909 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx"] Oct 30 14:17:09.058828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:09.058800 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236e8450-z2spx"] Oct 30 14:17:10.319273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:10.319241 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e9fad3-48c8-45f3-9346-4ff5bab94be9" path="/var/lib/kubelet/pods/f4e9fad3-48c8-45f3-9346-4ff5bab94be9/volumes" Oct 30 14:17:15.602488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.602451 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr"] Oct 30 14:17:15.603353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.603337 5824 cpu_manager.go:414] "containerMap: removing container" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" Oct 30 14:17:15.603409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.603354 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" Oct 30 14:17:15.603497 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.603487 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a4cb10-fc4e-4fa3-9ad9-977a6e5a499d" containerName="istio-proxy" Oct 30 14:17:15.608783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.608764 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.628867 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.628830 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr"] Oct 30 14:17:15.642041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.642010 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:17:15.642041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.642011 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:17:15.660599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.660570 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.660747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.660680 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpcl\" (UniqueName: \"kubernetes.io/projected/c7901f9c-b353-4468-ac78-904bdf79a107-kube-api-access-rkpcl\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.661120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.661093 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.762843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.762808 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.762996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.762894 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.762996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.762950 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpcl\" (UniqueName: \"kubernetes.io/projected/c7901f9c-b353-4468-ac78-904bdf79a107-kube-api-access-rkpcl\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.763655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.763631 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.765372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.765349 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.789999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.789973 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpcl\" (UniqueName: \"kubernetes.io/projected/c7901f9c-b353-4468-ac78-904bdf79a107-kube-api-access-rkpcl\") pod \"maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:15.930786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:15.930764 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:16.241106 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:17:16.241075 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7901f9c_b353_4468_ac78_904bdf79a107.slice/crio-0a59bbf0afc8b17c02ae952871342f60755eac50d839218564af4868984c207e WatchSource:0}: Error finding container 0a59bbf0afc8b17c02ae952871342f60755eac50d839218564af4868984c207e: Status 404 returned error can't find the container with id 0a59bbf0afc8b17c02ae952871342f60755eac50d839218564af4868984c207e Oct 30 14:17:16.258885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:16.258853 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr"] Oct 30 14:17:16.792699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:16.792672 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn"] Oct 30 14:17:16.797447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:16.797423 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:16.814981 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:16.814956 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d727dc-dockercfg-mfqgg\"" Oct 30 14:17:16.814981 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:16.814960 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d727dc\"" Oct 30 14:17:16.815137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:16.815053 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn"] Oct 30 14:17:16.934265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:16.925600 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxww\" (UniqueName: \"kubernetes.io/projected/84afee28-cab0-479b-9867-d2605054f28a-kube-api-access-qmxww\") pod \"payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn\" (UID: \"84afee28-cab0-479b-9867-d2605054f28a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:16.934265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:16.925942 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afee28-cab0-479b-9867-d2605054f28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn\" (UID: \"84afee28-cab0-479b-9867-d2605054f28a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:17.032020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.027567 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afee28-cab0-479b-9867-d2605054f28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn\" (UID: \"84afee28-cab0-479b-9867-d2605054f28a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:17.032020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.027741 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxww\" (UniqueName: \"kubernetes.io/projected/84afee28-cab0-479b-9867-d2605054f28a-kube-api-access-qmxww\") pod \"payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn\" (UID: \"84afee28-cab0-479b-9867-d2605054f28a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:17.032020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.029081 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afee28-cab0-479b-9867-d2605054f28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn\" (UID: \"84afee28-cab0-479b-9867-d2605054f28a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:17.048620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.048588 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f"] Oct 30 14:17:17.054115 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.054074 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:17.058776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.058628 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f"] Oct 30 14:17:17.063151 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.063123 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxww\" (UniqueName: \"kubernetes.io/projected/84afee28-cab0-479b-9867-d2605054f28a-kube-api-access-qmxww\") pod \"payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn\" (UID: \"84afee28-cab0-479b-9867-d2605054f28a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:17.107781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.104528 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" event={"ID":"c7901f9c-b353-4468-ac78-904bdf79a107","Type":"ContainerStarted","Data":"b026b2bab02b7138c72c09c0a4e7ec7ccec101015c7a1f7c54c5f57c69f17d5d"} Oct 30 14:17:17.107781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.104590 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" event={"ID":"c7901f9c-b353-4468-ac78-904bdf79a107","Type":"ContainerStarted","Data":"0a59bbf0afc8b17c02ae952871342f60755eac50d839218564af4868984c207e"} Oct 30 14:17:17.114153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.114125 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:17.177128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.177077 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" podStartSLOduration=2.177062895 podStartE2EDuration="2.177062895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:17.17401293 +0000 UTC m=+1666.974518374" watchObservedRunningTime="2026-10-30 14:17:17.177062895 +0000 UTC m=+1666.977568338" Oct 30 14:17:17.240386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.235064 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46lz\" (UniqueName: \"kubernetes.io/projected/578ef5f4-d95a-4384-965e-dc8561763917-kube-api-access-x46lz\") pod \"payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f\" (UID: \"578ef5f4-d95a-4384-965e-dc8561763917\") " pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:17.240386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.236075 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578ef5f4-d95a-4384-965e-dc8561763917-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f\" (UID: \"578ef5f4-d95a-4384-965e-dc8561763917\") " pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:17.338127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.338093 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578ef5f4-d95a-4384-965e-dc8561763917-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f\" (UID: \"578ef5f4-d95a-4384-965e-dc8561763917\") " pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:17.338360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.338346 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46lz\" (UniqueName: \"kubernetes.io/projected/578ef5f4-d95a-4384-965e-dc8561763917-kube-api-access-x46lz\") pod \"payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f\" (UID: \"578ef5f4-d95a-4384-965e-dc8561763917\") " pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:17.338798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.338777 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578ef5f4-d95a-4384-965e-dc8561763917-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f\" (UID: \"578ef5f4-d95a-4384-965e-dc8561763917\") " pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:17.364552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.364531 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46lz\" (UniqueName: \"kubernetes.io/projected/578ef5f4-d95a-4384-965e-dc8561763917-kube-api-access-x46lz\") pod \"payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f\" (UID: \"578ef5f4-d95a-4384-965e-dc8561763917\") " pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:17.414347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.414326 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:17.453737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.453706 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn"] Oct 30 14:17:17.727851 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:17:17.727822 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578ef5f4_d95a_4384_965e_dc8561763917.slice/crio-899dcc1b302e1a2ce2ada54c9ef068329a47b8c629a4117869ee9bdc4361a39d WatchSource:0}: Error finding container 899dcc1b302e1a2ce2ada54c9ef068329a47b8c629a4117869ee9bdc4361a39d: Status 404 returned error can't find the container with id 899dcc1b302e1a2ce2ada54c9ef068329a47b8c629a4117869ee9bdc4361a39d Oct 30 14:17:17.729797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:17.729685 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f"] Oct 30 14:17:18.114370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.114279 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" event={"ID":"578ef5f4-d95a-4384-965e-dc8561763917","Type":"ContainerStarted","Data":"d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f"} Oct 30 14:17:18.114370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.114335 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" event={"ID":"578ef5f4-d95a-4384-965e-dc8561763917","Type":"ContainerStarted","Data":"899dcc1b302e1a2ce2ada54c9ef068329a47b8c629a4117869ee9bdc4361a39d"} Oct 30 14:17:18.114839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.114379 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:18.116121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.116097 5824 generic.go:356] "Generic (PLEG): container finished" podID="c7901f9c-b353-4468-ac78-904bdf79a107" containerID="b026b2bab02b7138c72c09c0a4e7ec7ccec101015c7a1f7c54c5f57c69f17d5d" exitCode=0 Oct 30 14:17:18.116256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.116152 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" event={"ID":"c7901f9c-b353-4468-ac78-904bdf79a107","Type":"ContainerDied","Data":"b026b2bab02b7138c72c09c0a4e7ec7ccec101015c7a1f7c54c5f57c69f17d5d"} Oct 30 14:17:18.117950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.117926 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" event={"ID":"84afee28-cab0-479b-9867-d2605054f28a","Type":"ContainerStarted","Data":"11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41"} Oct 30 14:17:18.118062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.117972 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" event={"ID":"84afee28-cab0-479b-9867-d2605054f28a","Type":"ContainerStarted","Data":"f5e6bc0c2904c90376ba2a82a2b1c09097422ec20260d106243721abf124450a"} Oct 30 14:17:18.118062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.118027 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:18.161178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.161128 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" podStartSLOduration=2.161117466 podStartE2EDuration="2.161117466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:18.158309853 +0000 UTC m=+1667.958815297" watchObservedRunningTime="2026-10-30 14:17:18.161117466 +0000 UTC m=+1667.961622945" Oct 30 14:17:18.202843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:18.202798 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" podStartSLOduration=2.202784769 podStartE2EDuration="2.202784769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:18.20189604 +0000 UTC m=+1668.002401495" watchObservedRunningTime="2026-10-30 14:17:18.202784769 +0000 UTC m=+1668.003290218" Oct 30 14:17:19.294099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.294070 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:19.379329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.379239 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7901f9c-b353-4468-ac78-904bdf79a107-kube-api-access-rkpcl\" (UniqueName: \"kubernetes.io/projected/c7901f9c-b353-4468-ac78-904bdf79a107-kube-api-access-rkpcl\") pod \"c7901f9c-b353-4468-ac78-904bdf79a107\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " Oct 30 14:17:19.379515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.379440 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-openshift-ca\") pod \"c7901f9c-b353-4468-ac78-904bdf79a107\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " Oct 30 14:17:19.379723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.379700 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-xks-ca\") pod \"c7901f9c-b353-4468-ac78-904bdf79a107\" (UID: \"c7901f9c-b353-4468-ac78-904bdf79a107\") " Oct 30 14:17:19.380162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.380133 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-openshift-ca" pod "c7901f9c-b353-4468-ac78-904bdf79a107" (UID: "c7901f9c-b353-4468-ac78-904bdf79a107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:17:19.380500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.380474 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:19.383005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.382978 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7901f9c-b353-4468-ac78-904bdf79a107-kube-api-access-rkpcl" pod "c7901f9c-b353-4468-ac78-904bdf79a107" (UID: "c7901f9c-b353-4468-ac78-904bdf79a107"). InnerVolumeSpecName "kube-api-access-rkpcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:17:19.383138 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.383084 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-xks-ca" pod "c7901f9c-b353-4468-ac78-904bdf79a107" (UID: "c7901f9c-b353-4468-ac78-904bdf79a107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:17:19.482889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.482853 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7901f9c-b353-4468-ac78-904bdf79a107-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:19.482889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:19.482889 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkpcl\" (UniqueName: \"kubernetes.io/projected/c7901f9c-b353-4468-ac78-904bdf79a107-kube-api-access-rkpcl\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:20.135864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:20.135836 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" Oct 30 14:17:20.135864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:20.135843 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr" event={"ID":"c7901f9c-b353-4468-ac78-904bdf79a107","Type":"ContainerDied","Data":"0a59bbf0afc8b17c02ae952871342f60755eac50d839218564af4868984c207e"} Oct 30 14:17:20.136065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:20.135895 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a59bbf0afc8b17c02ae952871342f60755eac50d839218564af4868984c207e" Oct 30 14:17:22.352478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.352437 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78"] Oct 30 14:17:22.353482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.353458 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c7901f9c-b353-4468-ac78-904bdf79a107" containerName="revoke-keys" Oct 30 14:17:22.353482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.353479 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7901f9c-b353-4468-ac78-904bdf79a107" containerName="revoke-keys" Oct 30 14:17:22.353703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.353688 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7901f9c-b353-4468-ac78-904bdf79a107" containerName="revoke-keys" Oct 30 14:17:22.357408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.357387 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:22.376114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.376087 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78"] Oct 30 14:17:22.384786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.384761 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-972fee\"" Oct 30 14:17:22.425432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.425400 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wdj4\" (UniqueName: \"kubernetes.io/projected/b0edc37e-f5bf-48e8-a088-ee66527dd569-kube-api-access-9wdj4\") pod \"payload-processing-e2e-api-a-972fee-546d4b6686-svw78\" (UID: \"b0edc37e-f5bf-48e8-a088-ee66527dd569\") " pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:22.425695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.425496 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0edc37e-f5bf-48e8-a088-ee66527dd569-plugins-config-volume\") pod \"payload-processing-e2e-api-a-972fee-546d4b6686-svw78\" (UID: \"b0edc37e-f5bf-48e8-a088-ee66527dd569\") " pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:22.545211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.544902 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wdj4\" (UniqueName: \"kubernetes.io/projected/b0edc37e-f5bf-48e8-a088-ee66527dd569-kube-api-access-9wdj4\") pod \"payload-processing-e2e-api-a-972fee-546d4b6686-svw78\" (UID: \"b0edc37e-f5bf-48e8-a088-ee66527dd569\") " pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:22.545211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.544998 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0edc37e-f5bf-48e8-a088-ee66527dd569-plugins-config-volume\") pod \"payload-processing-e2e-api-a-972fee-546d4b6686-svw78\" (UID: \"b0edc37e-f5bf-48e8-a088-ee66527dd569\") " pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:22.558177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.557531 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0edc37e-f5bf-48e8-a088-ee66527dd569-plugins-config-volume\") pod \"payload-processing-e2e-api-a-972fee-546d4b6686-svw78\" (UID: \"b0edc37e-f5bf-48e8-a088-ee66527dd569\") " pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:22.594518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.594480 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wdj4\" (UniqueName: \"kubernetes.io/projected/b0edc37e-f5bf-48e8-a088-ee66527dd569-kube-api-access-9wdj4\") pod \"payload-processing-e2e-api-a-972fee-546d4b6686-svw78\" (UID: \"b0edc37e-f5bf-48e8-a088-ee66527dd569\") " pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:22.677527 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:22.677496 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:23.012401 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:17:23.012371 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0edc37e_f5bf_48e8_a088_ee66527dd569.slice/crio-b5132c91525cf2071d7342b381117904cd943308f7e6f9acf0cde1b5ea04256b WatchSource:0}: Error finding container b5132c91525cf2071d7342b381117904cd943308f7e6f9acf0cde1b5ea04256b: Status 404 returned error can't find the container with id b5132c91525cf2071d7342b381117904cd943308f7e6f9acf0cde1b5ea04256b Oct 30 14:17:23.013670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:23.013649 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78"] Oct 30 14:17:23.163606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:23.163567 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" event={"ID":"b0edc37e-f5bf-48e8-a088-ee66527dd569","Type":"ContainerStarted","Data":"a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac"} Oct 30 14:17:23.163777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:23.163633 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" event={"ID":"b0edc37e-f5bf-48e8-a088-ee66527dd569","Type":"ContainerStarted","Data":"b5132c91525cf2071d7342b381117904cd943308f7e6f9acf0cde1b5ea04256b"} Oct 30 14:17:23.163777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:23.163661 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:23.208537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:23.208487 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" podStartSLOduration=1.208468998 podStartE2EDuration="1.208468998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:23.20808838 +0000 UTC m=+1673.008593825" watchObservedRunningTime="2026-10-30 14:17:23.208468998 +0000 UTC m=+1673.008974444" Oct 30 14:17:26.121994 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:26.121956 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl"] Oct 30 14:17:29.128954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:29.128923 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:17:29.129360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:29.128996 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:17:30.922481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:30.922447 5824 scope.go:122] "RemoveContainer" containerID="9c8f06fa5e0000fca6cdb02494e49c81b412b431ba1a8dd33eba70a96b587d33" Oct 30 14:17:30.939219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:30.939180 5824 scope.go:122] "RemoveContainer" containerID="ebdf343821e36bfd899609add151bc63c38468390252a3e9d005c6e57c37e269" Oct 30 14:17:30.955893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:30.955862 5824 scope.go:122] "RemoveContainer" containerID="4c49634f1813de328b1d342b042a43d741656c3a6c75d7ecc5170a49d77c0d10" Oct 30 14:17:31.122537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:31.122486 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" podUID="9de68d4e-71ca-49be-ba3f-0ece4545245e" containerName="payload-pre-processing" containerID="cri-o://a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7" gracePeriod=25 Oct 30 14:17:32.980424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:32.980384 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" podUID="9de68d4e-71ca-49be-ba3f-0ece4545245e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:17:33.642471 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:17:33.642423 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:17:34.183112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:34.183079 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:17:38.048743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:38.048706 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66"] Oct 30 14:17:38.061828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:38.061799 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f2bd82-n7w66"] Oct 30 14:17:38.328181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:38.328095 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7518f76-e056-4ce2-8462-f693ad20cedf" path="/var/lib/kubelet/pods/b7518f76-e056-4ce2-8462-f693ad20cedf/volumes" Oct 30 14:17:42.980414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:42.980370 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" podUID="9de68d4e-71ca-49be-ba3f-0ece4545245e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:17:44.179307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.179269 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c"] Oct 30 14:17:44.187723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.187694 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:44.210425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.210395 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c"] Oct 30 14:17:44.215582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.215550 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d74808-dockercfg-77sdb\"" Oct 30 14:17:44.215745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.215616 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d74808\"" Oct 30 14:17:44.394686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.394642 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf"] Oct 30 14:17:44.397001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.396964 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c\" (UID: \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:44.397174 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.397134 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lg8\" (UniqueName: \"kubernetes.io/projected/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-kube-api-access-p9lg8\") pod \"payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c\" (UID: \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:44.411671 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.411640 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:44.417391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.417365 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf"] Oct 30 14:17:44.499360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.499273 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk76f\" (UniqueName: \"kubernetes.io/projected/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-kube-api-access-nk76f\") pod \"payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf\" (UID: \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:44.499518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.499432 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf\" (UID: \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:44.499573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.499547 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c\" (UID: \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:44.499689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.499629 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lg8\" (UniqueName: \"kubernetes.io/projected/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-kube-api-access-p9lg8\") pod \"payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c\" (UID: \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:44.500350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.500323 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c\" (UID: \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:44.525875 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.525845 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lg8\" (UniqueName: \"kubernetes.io/projected/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-kube-api-access-p9lg8\") pod \"payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c\" (UID: \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:44.601468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.601429 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk76f\" (UniqueName: \"kubernetes.io/projected/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-kube-api-access-nk76f\") pod \"payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf\" (UID: \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:44.601677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.601557 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf\" (UID: \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:44.602329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.602308 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf\" (UID: \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:44.651130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.650939 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk76f\" (UniqueName: \"kubernetes.io/projected/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-kube-api-access-nk76f\") pod \"payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf\" (UID: \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:44.746212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.746039 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:44.818264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:44.818238 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:45.089150 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:17:45.089122 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1fb9bc_41d4_46e2_b760_4c370821dd4e.slice/crio-6b56f57f851e778167a43f3ded2f956041ec509b15ef259aa5491384e288193f WatchSource:0}: Error finding container 6b56f57f851e778167a43f3ded2f956041ec509b15ef259aa5491384e288193f: Status 404 returned error can't find the container with id 6b56f57f851e778167a43f3ded2f956041ec509b15ef259aa5491384e288193f Oct 30 14:17:45.090438 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.090412 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf"] Oct 30 14:17:45.166903 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:17:45.166873 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33d70fd_8ff6_4d54_8b1d_0dae28c945d5.slice/crio-69935234a923045ec0613587201bbea138678d53421f16d609178aaf973005f3 WatchSource:0}: Error finding container 69935234a923045ec0613587201bbea138678d53421f16d609178aaf973005f3: Status 404 returned error can't find the container with id 69935234a923045ec0613587201bbea138678d53421f16d609178aaf973005f3 Oct 30 14:17:45.168384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.168359 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c"] Oct 30 14:17:45.391491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.391395 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" event={"ID":"2a1fb9bc-41d4-46e2-b760-4c370821dd4e","Type":"ContainerStarted","Data":"c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88"} Oct 30 14:17:45.391491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.391457 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" event={"ID":"2a1fb9bc-41d4-46e2-b760-4c370821dd4e","Type":"ContainerStarted","Data":"6b56f57f851e778167a43f3ded2f956041ec509b15ef259aa5491384e288193f"} Oct 30 14:17:45.391985 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.391597 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:45.393125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.393103 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" event={"ID":"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5","Type":"ContainerStarted","Data":"ac3152cb7349407cb85b0b5fdab3a1ecb1bb6f34e1d05619f5fcad1038257361"} Oct 30 14:17:45.393245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.393147 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" event={"ID":"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5","Type":"ContainerStarted","Data":"69935234a923045ec0613587201bbea138678d53421f16d609178aaf973005f3"} Oct 30 14:17:45.393245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.393177 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:45.438125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.438083 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" podStartSLOduration=1.43807108 podStartE2EDuration="1.43807108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:45.436006915 +0000 UTC m=+1695.236512370" watchObservedRunningTime="2026-10-30 14:17:45.43807108 +0000 UTC m=+1695.238576554" Oct 30 14:17:45.482085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:45.482040 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" podStartSLOduration=1.482026409 podStartE2EDuration="1.482026409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:45.480096202 +0000 UTC m=+1695.280601646" watchObservedRunningTime="2026-10-30 14:17:45.482026409 +0000 UTC m=+1695.282531854" Oct 30 14:17:51.440646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.440611 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:17:51.446791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.446762 5824 generic.go:356] "Generic (PLEG): container finished" podID="9de68d4e-71ca-49be-ba3f-0ece4545245e" containerID="a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7" exitCode=0 Oct 30 14:17:51.446924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.446817 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" Oct 30 14:17:51.446924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.446856 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" event={"ID":"9de68d4e-71ca-49be-ba3f-0ece4545245e","Type":"ContainerDied","Data":"a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7"} Oct 30 14:17:51.446924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.446914 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl" event={"ID":"9de68d4e-71ca-49be-ba3f-0ece4545245e","Type":"ContainerDied","Data":"550d3419c4868d48791b2caa94b28e845a880e70dcad99eee32d151a1e023447"} Oct 30 14:17:51.447055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.446938 5824 scope.go:122] "RemoveContainer" containerID="a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7" Oct 30 14:17:51.458573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.458557 5824 scope.go:122] "RemoveContainer" containerID="a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7" Oct 30 14:17:51.458828 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:17:51.458804 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7\": container with ID starting with a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7 not found: ID does not exist" containerID="a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7" Oct 30 14:17:51.458870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.458836 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7"} err="failed to get container status \"a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7\": rpc error: code = NotFound desc = could not find container \"a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7\": container with ID starting with a6a01e56b6d50aa719e193c9456506cab6da79b58baabf2b4a9c79ecc99dcba7 not found: ID does not exist" Oct 30 14:17:51.520275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.520244 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de68d4e-71ca-49be-ba3f-0ece4545245e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de68d4e-71ca-49be-ba3f-0ece4545245e-plugins-config-volume\") pod \"9de68d4e-71ca-49be-ba3f-0ece4545245e\" (UID: \"9de68d4e-71ca-49be-ba3f-0ece4545245e\") " Oct 30 14:17:51.520444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.520430 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de68d4e-71ca-49be-ba3f-0ece4545245e-kube-api-access-qslt9\" (UniqueName: \"kubernetes.io/projected/9de68d4e-71ca-49be-ba3f-0ece4545245e-kube-api-access-qslt9\") pod \"9de68d4e-71ca-49be-ba3f-0ece4545245e\" (UID: \"9de68d4e-71ca-49be-ba3f-0ece4545245e\") " Oct 30 14:17:51.520721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.520684 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de68d4e-71ca-49be-ba3f-0ece4545245e-plugins-config-volume" pod "9de68d4e-71ca-49be-ba3f-0ece4545245e" (UID: "9de68d4e-71ca-49be-ba3f-0ece4545245e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:17:51.521286 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.521262 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de68d4e-71ca-49be-ba3f-0ece4545245e-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:51.522580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.522558 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de68d4e-71ca-49be-ba3f-0ece4545245e-kube-api-access-qslt9" pod "9de68d4e-71ca-49be-ba3f-0ece4545245e" (UID: "9de68d4e-71ca-49be-ba3f-0ece4545245e"). InnerVolumeSpecName "kube-api-access-qslt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:17:51.623950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.623914 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qslt9\" (UniqueName: \"kubernetes.io/projected/9de68d4e-71ca-49be-ba3f-0ece4545245e-kube-api-access-qslt9\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:17:51.862222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.855419 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl"] Oct 30 14:17:51.887211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:51.877258 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9dba4af9-7f54cffb47-6kdzl"] Oct 30 14:17:52.320695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:52.320662 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de68d4e-71ca-49be-ba3f-0ece4545245e" path="/var/lib/kubelet/pods/9de68d4e-71ca-49be-ba3f-0ece4545245e/volumes" Oct 30 14:17:56.404959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:56.404926 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:17:56.405892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:56.405831 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:17:56.408936 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:56.408914 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp"] Oct 30 14:17:56.409118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:56.409082 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" containerID="cri-o://8924a227f802e4348e80a2d26f06a1f5f3a22745b4ea46708c5b7e2e666b3159" gracePeriod=30 Oct 30 14:17:56.838966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:56.838935 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr"] Oct 30 14:17:56.839254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:56.839230 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" containerID="cri-o://a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c" gracePeriod=30 Oct 30 14:17:57.803050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.803004 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq"] Oct 30 14:17:57.804129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.804103 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9de68d4e-71ca-49be-ba3f-0ece4545245e" containerName="payload-pre-processing" Oct 30 14:17:57.804129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.804128 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de68d4e-71ca-49be-ba3f-0ece4545245e" containerName="payload-pre-processing" Oct 30 14:17:57.804413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.804396 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de68d4e-71ca-49be-ba3f-0ece4545245e" containerName="payload-pre-processing" Oct 30 14:17:57.810392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.810374 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq"] Oct 30 14:17:57.810507 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.810488 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:57.828747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.828718 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2b206b\"" Oct 30 14:17:57.883792 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.883767 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2b206b-dockercfg-rl7pf\"" Oct 30 14:17:57.980654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.980616 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b624cf-d772-4725-bad1-74d15fe3626f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq\" (UID: \"c7b624cf-d772-4725-bad1-74d15fe3626f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:57.980954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:57.980935 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jr8\" (UniqueName: \"kubernetes.io/projected/c7b624cf-d772-4725-bad1-74d15fe3626f-kube-api-access-c9jr8\") pod \"payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq\" (UID: \"c7b624cf-d772-4725-bad1-74d15fe3626f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:58.082255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:58.082127 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jr8\" (UniqueName: \"kubernetes.io/projected/c7b624cf-d772-4725-bad1-74d15fe3626f-kube-api-access-c9jr8\") pod \"payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq\" (UID: \"c7b624cf-d772-4725-bad1-74d15fe3626f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:58.082430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:58.082336 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b624cf-d772-4725-bad1-74d15fe3626f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq\" (UID: \"c7b624cf-d772-4725-bad1-74d15fe3626f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:58.083044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:58.083017 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b624cf-d772-4725-bad1-74d15fe3626f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq\" (UID: \"c7b624cf-d772-4725-bad1-74d15fe3626f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:58.108833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:58.108807 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jr8\" (UniqueName: \"kubernetes.io/projected/c7b624cf-d772-4725-bad1-74d15fe3626f-kube-api-access-c9jr8\") pod \"payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq\" (UID: \"c7b624cf-d772-4725-bad1-74d15fe3626f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:58.132228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:58.132183 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:58.440082 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:17:58.440056 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b624cf_d772_4725_bad1_74d15fe3626f.slice/crio-631a25c104fe459e169ddfe9ef7e2e012132f0821721f256b4039a10ce9f1daf WatchSource:0}: Error finding container 631a25c104fe459e169ddfe9ef7e2e012132f0821721f256b4039a10ce9f1daf: Status 404 returned error can't find the container with id 631a25c104fe459e169ddfe9ef7e2e012132f0821721f256b4039a10ce9f1daf Oct 30 14:17:58.443679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:58.443646 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq"] Oct 30 14:17:58.516410 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:58.516382 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" event={"ID":"c7b624cf-d772-4725-bad1-74d15fe3626f","Type":"ContainerStarted","Data":"631a25c104fe459e169ddfe9ef7e2e012132f0821721f256b4039a10ce9f1daf"} Oct 30 14:17:59.525729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:59.525689 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" event={"ID":"c7b624cf-d772-4725-bad1-74d15fe3626f","Type":"ContainerStarted","Data":"fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc"} Oct 30 14:17:59.526266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:59.525777 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:17:59.570261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:17:59.570183 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" podStartSLOduration=2.570169291 podStartE2EDuration="2.570169291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:17:59.570099149 +0000 UTC m=+1709.370604606" watchObservedRunningTime="2026-10-30 14:17:59.570169291 +0000 UTC m=+1709.370674734" Oct 30 14:18:01.458047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.458017 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 14:18:01.458370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.458067 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 14:18:01.545407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.545381 5824 generic.go:356] "Generic (PLEG): container finished" podID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerID="8924a227f802e4348e80a2d26f06a1f5f3a22745b4ea46708c5b7e2e666b3159" exitCode=0 Oct 30 14:18:01.545542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.545453 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" event={"ID":"3c6636d3-8077-4e87-95a3-e1325f6f677c","Type":"ContainerDied","Data":"8924a227f802e4348e80a2d26f06a1f5f3a22745b4ea46708c5b7e2e666b3159"} Oct 30 14:18:01.709026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.709001 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:18:01.748958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.748928 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-envoy\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.749070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.748967 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-socket\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.749147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749129 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-podinfo\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.749281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749183 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6636d3-8077-4e87-95a3-e1325f6f677c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6636d3-8077-4e87-95a3-e1325f6f677c-istiod-ca-cert\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.749343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749292 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-credential-socket\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.749343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749322 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-socket" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.749453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749347 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-kube-api-access-zk5g2\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-kube-api-access-zk5g2\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.749453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749379 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-token\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-token\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.749453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749426 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-data\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.749617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749466 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-certs\") pod \"3c6636d3-8077-4e87-95a3-e1325f6f677c\" (UID: \"3c6636d3-8077-4e87-95a3-e1325f6f677c\") " Oct 30 14:18:01.750003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.749971 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6636d3-8077-4e87-95a3-e1325f6f677c-istiod-ca-cert" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:01.750446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.750254 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.750446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.750282 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6636d3-8077-4e87-95a3-e1325f6f677c-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.750446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.750303 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-data" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.752851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.752826 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-envoy" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.753020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.752874 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-kube-api-access-zk5g2" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "kube-api-access-zk5g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:01.753104 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.753082 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-token" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:01.753177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.753141 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-certs" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.753354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.753335 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-credential-socket" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:01.770850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.769789 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-podinfo" pod "3c6636d3-8077-4e87-95a3-e1325f6f677c" (UID: "3c6636d3-8077-4e87-95a3-e1325f6f677c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:18:01.851407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.851376 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.851407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.851403 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.851407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.851413 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk5g2\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-kube-api-access-zk5g2\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.851623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.851425 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.851623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.851434 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.851623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.851442 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:01.851623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:01.851449 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6636d3-8077-4e87-95a3-e1325f6f677c-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.088737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.088710 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:18:02.156227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156171 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-certs\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.156410 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156314 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-credential-socket\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.156410 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156402 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istiod-ca-cert\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.156541 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156448 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-token\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-token\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.156541 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156491 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-podinfo\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.156541 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156489 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-certs" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:02.156698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156682 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-socket\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.156787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156765 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-envoy\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.156846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156803 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-data\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.156904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156858 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istiod-ca-cert" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:02.156962 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.156920 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-kube-api-access-8s6z7\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-kube-api-access-8s6z7\") pod \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\" (UID: \"3c2336d1-a75c-4870-a2ba-918cd6662c6e\") " Oct 30 14:18:02.157171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.157142 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-credential-socket" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:02.157724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.157698 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-data" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:02.157827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.157755 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-socket" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:02.157949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.157929 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.158020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.157956 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.158020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.157972 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.158020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.157987 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.160326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.160287 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-podinfo" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:18:02.160849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.160796 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-token" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:02.160849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.160184 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-kube-api-access-8s6z7" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "kube-api-access-8s6z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:02.161817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.161791 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-envoy" pod "3c2336d1-a75c-4870-a2ba-918cd6662c6e" (UID: "3c2336d1-a75c-4870-a2ba-918cd6662c6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:02.259506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.259431 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.259506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.259469 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s6z7\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-kube-api-access-8s6z7\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.259506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.259485 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.259506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.259500 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2336d1-a75c-4870-a2ba-918cd6662c6e-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.259746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.259513 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2336d1-a75c-4870-a2ba-918cd6662c6e-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:02.556443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.556360 5824 generic.go:356] "Generic (PLEG): container finished" podID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerID="a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c" exitCode=0 Oct 30 14:18:02.556829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.556444 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" Oct 30 14:18:02.556829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.556447 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" event={"ID":"3c2336d1-a75c-4870-a2ba-918cd6662c6e","Type":"ContainerDied","Data":"a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c"} Oct 30 14:18:02.556829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.556625 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr" event={"ID":"3c2336d1-a75c-4870-a2ba-918cd6662c6e","Type":"ContainerDied","Data":"cd18291e418426a499d4a596cbc5e1687e8a90944ba3a10e137d9bd4f6296713"} Oct 30 14:18:02.556829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.556665 5824 scope.go:122] "RemoveContainer" containerID="a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c" Oct 30 14:18:02.558152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.558132 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" Oct 30 14:18:02.558303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.558161 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp" event={"ID":"3c6636d3-8077-4e87-95a3-e1325f6f677c","Type":"ContainerDied","Data":"c05f3143718a84d695d7c5b40c7bd7576d33490e28d7a57dab06ad6f7e469f45"} Oct 30 14:18:02.571753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.571735 5824 scope.go:122] "RemoveContainer" containerID="a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c" Oct 30 14:18:02.572026 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:18:02.572007 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c\": container with ID starting with a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c not found: ID does not exist" containerID="a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c" Oct 30 14:18:02.572073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.572037 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c"} err="failed to get container status \"a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c\": rpc error: code = NotFound desc = could not find container \"a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c\": container with ID starting with a267dc010523601b4e106b6c9d0bd91f1dcbb297d008dad349b5cdfaa78cfd6c not found: ID does not exist" Oct 30 14:18:02.572073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.572056 5824 scope.go:122] "RemoveContainer" containerID="8924a227f802e4348e80a2d26f06a1f5f3a22745b4ea46708c5b7e2e666b3159" Oct 30 14:18:02.627530 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.627466 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr"] Oct 30 14:18:02.636461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.635793 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-mfhnr"] Oct 30 14:18:02.675798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.667257 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp"] Oct 30 14:18:02.675798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:02.673146 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-cmxbp"] Oct 30 14:18:03.642060 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:18:03.642025 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:18:04.319006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.318971 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" path="/var/lib/kubelet/pods/3c2336d1-a75c-4870-a2ba-918cd6662c6e/volumes" Oct 30 14:18:04.319515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.319499 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" path="/var/lib/kubelet/pods/3c6636d3-8077-4e87-95a3-e1325f6f677c/volumes" Oct 30 14:18:04.894225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.894167 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6"] Oct 30 14:18:04.895062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.895044 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" Oct 30 14:18:04.895062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.895062 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" Oct 30 14:18:04.895151 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.895094 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" Oct 30 14:18:04.895151 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.895099 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" Oct 30 14:18:04.895248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.895235 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2336d1-a75c-4870-a2ba-918cd6662c6e" containerName="istio-proxy" Oct 30 14:18:04.895288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.895254 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6636d3-8077-4e87-95a3-e1325f6f677c" containerName="istio-proxy" Oct 30 14:18:04.900842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.900823 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:04.917086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.917060 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6"] Oct 30 14:18:04.925761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.925633 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:18:04.925761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:04.925702 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:18:05.004840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.004814 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqw86\" (UniqueName: \"kubernetes.io/projected/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-kube-api-access-pqw86\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.004978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.004864 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.005077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.005056 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.106560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.106532 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqw86\" (UniqueName: \"kubernetes.io/projected/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-kube-api-access-pqw86\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.106685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.106572 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.106685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.106658 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.107387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.107363 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.108931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.108905 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.133628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.133599 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqw86\" (UniqueName: \"kubernetes.io/projected/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-kube-api-access-pqw86\") pod \"maas-api-revoke-sub-08ffca3b-8nlp6\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.218291 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.218272 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:05.529590 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:18:05.529558 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb0d62e_e0a6_4be1_a1d8_3cf02cce615c.slice/crio-c7c52b0f1c6fe35c684639dfac8751cb36e124cbbfd33d92f6164fe5ee08382f WatchSource:0}: Error finding container c7c52b0f1c6fe35c684639dfac8751cb36e124cbbfd33d92f6164fe5ee08382f: Status 404 returned error can't find the container with id c7c52b0f1c6fe35c684639dfac8751cb36e124cbbfd33d92f6164fe5ee08382f Oct 30 14:18:05.531177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.531149 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6"] Oct 30 14:18:05.587276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:05.587243 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" event={"ID":"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c","Type":"ContainerStarted","Data":"c7c52b0f1c6fe35c684639dfac8751cb36e124cbbfd33d92f6164fe5ee08382f"} Oct 30 14:18:06.597470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:06.597437 5824 generic.go:356] "Generic (PLEG): container finished" podID="6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c" containerID="ce999ea31be768fe82505b4a007569e047d7e696fcda25335580e51261f759ff" exitCode=0 Oct 30 14:18:06.597789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:06.597569 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" event={"ID":"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c","Type":"ContainerDied","Data":"ce999ea31be768fe82505b4a007569e047d7e696fcda25335580e51261f759ff"} Oct 30 14:18:07.801831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.801796 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:07.861072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.860996 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-xks-ca\") pod \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " Oct 30 14:18:07.861235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.861168 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-openshift-ca\") pod \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " Oct 30 14:18:07.861581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.861552 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-openshift-ca" pod "6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c" (UID: "6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:07.861750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.861572 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-kube-api-access-pqw86\" (UniqueName: \"kubernetes.io/projected/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-kube-api-access-pqw86\") pod \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\" (UID: \"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c\") " Oct 30 14:18:07.862704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.862677 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:07.863768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.863740 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-xks-ca" pod "6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c" (UID: "6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:18:07.863871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.863846 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-kube-api-access-pqw86" pod "6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c" (UID: "6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c"). InnerVolumeSpecName "kube-api-access-pqw86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:07.969140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.967691 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqw86\" (UniqueName: \"kubernetes.io/projected/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-kube-api-access-pqw86\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:07.969140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:07.967734 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:08.617019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:08.616936 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" Oct 30 14:18:08.617019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:08.616942 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6" event={"ID":"6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c","Type":"ContainerDied","Data":"c7c52b0f1c6fe35c684639dfac8751cb36e124cbbfd33d92f6164fe5ee08382f"} Oct 30 14:18:08.617019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:08.616991 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7c52b0f1c6fe35c684639dfac8751cb36e124cbbfd33d92f6164fe5ee08382f" Oct 30 14:18:10.538665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:10.538637 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:18:14.499894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:14.499800 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr"] Oct 30 14:18:14.514512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:14.514482 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9dba4af9-w2nxr"] Oct 30 14:18:16.320782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:16.320751 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7901f9c-b353-4468-ac78-904bdf79a107" path="/var/lib/kubelet/pods/c7901f9c-b353-4468-ac78-904bdf79a107/volumes" Oct 30 14:18:26.425617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:26.425577 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb"] Oct 30 14:18:26.426089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:26.425849 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" containerID="cri-o://e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa" gracePeriod=30 Oct 30 14:18:31.079290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.079258 5824 scope.go:122] "RemoveContainer" containerID="d031d17858449854e8ff5d7f516df2b37c5569084f818fc1ef3162884cfb3528" Oct 30 14:18:31.090401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.090364 5824 scope.go:122] "RemoveContainer" containerID="881b34ec0a84e41cc73df3bdcc8bfbd688750fba9f086d5a619bdde782677746" Oct 30 14:18:31.102152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.102130 5824 scope.go:122] "RemoveContainer" containerID="c65a9c6de34a80051b48c9798ebc6c83cca048996f5f03dc32e97d72c7972e1e" Oct 30 14:18:31.111327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.111302 5824 scope.go:122] "RemoveContainer" containerID="9f682971ef0f8fdb35b4d2ee1fc2f65dfe254806407009f2d8c0ce73ebe93e30" Oct 30 14:18:31.808219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.795306 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:18:31.900794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.900752 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-envoy\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.900999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.900958 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istiod-ca-cert\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.901614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.901170 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-podinfo\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.901614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.901601 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istiod-ca-cert" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:31.901825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.901654 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-socket\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.901825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.901729 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-credential-socket\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.901825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.901759 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-certs\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.902008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.901824 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-token\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-token\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.902008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.901888 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-kube-api-access-fm94d\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-kube-api-access-fm94d\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.902008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.901919 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-data\") pod \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\" (UID: \"81a0a566-f79f-4fb4-b5be-9d2862d656a5\") " Oct 30 14:18:31.903537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.903497 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-credential-socket" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.903769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.903730 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-socket" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.904144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.904117 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-data" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.905461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.905438 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-certs" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.906359 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.906336 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.906518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.906500 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.906632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.906618 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.906734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.906720 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.906857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.906846 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:31.907576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.907534 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-token" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:31.908088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.908004 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-envoy" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:31.909622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.909587 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-kube-api-access-fm94d" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "kube-api-access-fm94d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:31.909710 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.909654 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-podinfo" pod "81a0a566-f79f-4fb4-b5be-9d2862d656a5" (UID: "81a0a566-f79f-4fb4-b5be-9d2862d656a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:18:31.965638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.965555 5824 generic.go:356] "Generic (PLEG): container finished" podID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerID="e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa" exitCode=0 Oct 30 14:18:31.965638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.965617 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" Oct 30 14:18:31.965828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.965633 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" event={"ID":"81a0a566-f79f-4fb4-b5be-9d2862d656a5","Type":"ContainerDied","Data":"e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa"} Oct 30 14:18:31.965828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.965694 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb" event={"ID":"81a0a566-f79f-4fb4-b5be-9d2862d656a5","Type":"ContainerDied","Data":"b4f15f041cd9e2e1b48ebd887479fd1af2bc2d3d9dccecfac258a690387ed017"} Oct 30 14:18:31.965828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.965731 5824 scope.go:122] "RemoveContainer" containerID="e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa" Oct 30 14:18:31.980091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.980067 5824 scope.go:122] "RemoveContainer" containerID="e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa" Oct 30 14:18:31.980421 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:18:31.980402 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa\": container with ID starting with e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa not found: ID does not exist" containerID="e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa" Oct 30 14:18:31.980498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:31.980428 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa"} err="failed to get container status \"e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa\": rpc error: code = NotFound desc = could not find container \"e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa\": container with ID starting with e26370fa2538268085c7c51cdb4d203869b3703aa5612a37510e54aaeeb3b0fa not found: ID does not exist" Oct 30 14:18:32.008518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:32.008490 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:32.008518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:32.008519 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:32.008676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:32.008534 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm94d\" (UniqueName: \"kubernetes.io/projected/81a0a566-f79f-4fb4-b5be-9d2862d656a5-kube-api-access-fm94d\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:32.008676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:32.008547 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a0a566-f79f-4fb4-b5be-9d2862d656a5-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:32.036597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:32.036572 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb"] Oct 30 14:18:32.042171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:32.042147 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-gphbb"] Oct 30 14:18:32.397408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:32.397323 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" path="/var/lib/kubelet/pods/81a0a566-f79f-4fb4-b5be-9d2862d656a5/volumes" Oct 30 14:18:33.709417 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:18:33.709372 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:18:41.506569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:41.506533 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs"] Oct 30 14:18:41.506975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:41.506748 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" containerID="cri-o://c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c" gracePeriod=30 Oct 30 14:18:41.580303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:41.580271 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj"] Oct 30 14:18:41.580491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:41.580470 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" containerID="cri-o://afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833" gracePeriod=30 Oct 30 14:18:46.896844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.896809 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:18:46.976428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.976400 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:18:46.986690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.986656 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-kube-api-access-dkstc\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-kube-api-access-dkstc\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.986835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.986743 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-data\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.986835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.986781 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c747c2-f178-4a1c-a666-634e8bc57fef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c747c2-f178-4a1c-a666-634e8bc57fef-istiod-ca-cert\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.987071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987050 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-socket\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.987156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987133 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-credential-socket\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.987274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987169 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-certs\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.987274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987220 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c747c2-f178-4a1c-a666-634e8bc57fef-istiod-ca-cert" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:46.987274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987248 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-token\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-token\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.987483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987307 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-envoy\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.987483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987433 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-podinfo\") pod \"55c747c2-f178-4a1c-a666-634e8bc57fef\" (UID: \"55c747c2-f178-4a1c-a666-634e8bc57fef\") " Oct 30 14:18:46.987483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987439 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-socket" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:46.987793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987769 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-certs" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:46.987888 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.987797 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-credential-socket" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:46.988152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.988130 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-data" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:46.988386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.988329 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:46.988386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.988352 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:46.988386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.988366 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:46.988386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.988379 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c747c2-f178-4a1c-a666-634e8bc57fef-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:46.988548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:46.988391 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.004514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.004446 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-kube-api-access-dkstc" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "kube-api-access-dkstc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:47.004893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.004851 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-envoy" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.005349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.005305 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-podinfo" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:18:47.005559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.005530 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-token" pod "55c747c2-f178-4a1c-a666-634e8bc57fef" (UID: "55c747c2-f178-4a1c-a666-634e8bc57fef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:47.089095 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089009 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-data\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089100 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-credential-socket\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089167 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istiod-ca-cert\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089268 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-socket\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089410 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-token\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-token\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089527 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-credential-socket" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.089603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089587 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-kube-api-access-rzqkp\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-kube-api-access-rzqkp\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089630 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-podinfo\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089699 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-certs\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089752 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089741 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-envoy\") pod \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\" (UID: \"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735\") " Oct 30 14:18:47.089910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.089886 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istiod-ca-cert" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:47.090288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.090261 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-data" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.090676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.090654 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.090756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.090684 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.090756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.090699 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.090756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.090714 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.090756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.090726 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.090756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.090740 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c747c2-f178-4a1c-a666-634e8bc57fef-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.090756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.090753 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkstc\" (UniqueName: \"kubernetes.io/projected/55c747c2-f178-4a1c-a666-634e8bc57fef-kube-api-access-dkstc\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.091162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.091135 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-socket" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.091387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.091367 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-certs" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.092970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.092942 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-podinfo" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:18:47.093079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.092975 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-envoy" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:18:47.093633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.093608 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-token" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:47.096371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.096346 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-kube-api-access-rzqkp" pod "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" (UID: "a3c14891-26f7-43e9-8c8a-ff9e5fc3c735"). InnerVolumeSpecName "kube-api-access-rzqkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:47.110629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.110600 5824 generic.go:356] "Generic (PLEG): container finished" podID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerID="afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833" exitCode=0 Oct 30 14:18:47.110750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.110662 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" Oct 30 14:18:47.110750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.110683 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" event={"ID":"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735","Type":"ContainerDied","Data":"afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833"} Oct 30 14:18:47.110750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.110739 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj" event={"ID":"a3c14891-26f7-43e9-8c8a-ff9e5fc3c735","Type":"ContainerDied","Data":"576b94774c13abf1b98e35e944db29f2904fcd762eff6ea1938688dbf327f116"} Oct 30 14:18:47.110930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.110764 5824 scope.go:122] "RemoveContainer" containerID="afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833" Oct 30 14:18:47.113663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.113635 5824 generic.go:356] "Generic (PLEG): container finished" podID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerID="c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c" exitCode=0 Oct 30 14:18:47.113756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.113706 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" Oct 30 14:18:47.113816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.113704 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" event={"ID":"55c747c2-f178-4a1c-a666-634e8bc57fef","Type":"ContainerDied","Data":"c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c"} Oct 30 14:18:47.113872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.113841 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs" event={"ID":"55c747c2-f178-4a1c-a666-634e8bc57fef","Type":"ContainerDied","Data":"74d4cb26083ba281e0626d03183aff0a6d59635f279e0209f3c582bd1b51f141"} Oct 30 14:18:47.124182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.124159 5824 scope.go:122] "RemoveContainer" containerID="afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833" Oct 30 14:18:47.124495 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:18:47.124478 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833\": container with ID starting with afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833 not found: ID does not exist" containerID="afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833" Oct 30 14:18:47.124568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.124501 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833"} err="failed to get container status \"afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833\": rpc error: code = NotFound desc = could not find container \"afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833\": container with ID starting with afcf07a916ecc50382c976cf9fba7234742d0d0d95551fc013470a343819d833 not found: ID does not exist" Oct 30 14:18:47.124568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.124519 5824 scope.go:122] "RemoveContainer" containerID="c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c" Oct 30 14:18:47.139891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.139863 5824 scope.go:122] "RemoveContainer" containerID="c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c" Oct 30 14:18:47.140446 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:18:47.140422 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c\": container with ID starting with c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c not found: ID does not exist" containerID="c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c" Oct 30 14:18:47.140568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.140452 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c"} err="failed to get container status \"c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c\": rpc error: code = NotFound desc = could not find container \"c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c\": container with ID starting with c02e44354c8a6424d3e4eba4bcf7e59649720652dca34daa3a1da53c5599641c not found: ID does not exist" Oct 30 14:18:47.177584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.177550 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj"] Oct 30 14:18:47.181417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.181389 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-f6tdj"] Oct 30 14:18:47.192396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.192373 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.192396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.192396 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzqkp\" (UniqueName: \"kubernetes.io/projected/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-kube-api-access-rzqkp\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.192539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.192406 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.192539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.192416 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.192539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.192425 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.192539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.192434 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:47.234148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.234121 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs"] Oct 30 14:18:47.240548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:47.240518 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-q5vhs"] Oct 30 14:18:48.444577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:48.444541 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" path="/var/lib/kubelet/pods/55c747c2-f178-4a1c-a666-634e8bc57fef/volumes" Oct 30 14:18:48.447540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:48.447516 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" path="/var/lib/kubelet/pods/a3c14891-26f7-43e9-8c8a-ff9e5fc3c735/volumes" Oct 30 14:18:53.977816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:53.977777 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8"] Oct 30 14:18:54.012283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:54.012257 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq"] Oct 30 14:18:58.978511 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:58.978470 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" podUID="6507c54e-2da9-4fb6-9014-14da28aa9ec8" containerName="payload-processing" containerID="cri-o://46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364" gracePeriod=25 Oct 30 14:18:59.264674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.264648 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:18:59.266573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.266548 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" podUID="adec2d68-9246-43fe-9c98-22fbb8891fb6" containerName="payload-pre-processing" containerID="cri-o://92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997" gracePeriod=25 Oct 30 14:18:59.308253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.308221 5824 generic.go:356] "Generic (PLEG): container finished" podID="6507c54e-2da9-4fb6-9014-14da28aa9ec8" containerID="46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364" exitCode=0 Oct 30 14:18:59.308377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.308323 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" event={"ID":"6507c54e-2da9-4fb6-9014-14da28aa9ec8","Type":"ContainerDied","Data":"46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364"} Oct 30 14:18:59.308444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.308376 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" event={"ID":"6507c54e-2da9-4fb6-9014-14da28aa9ec8","Type":"ContainerDied","Data":"8c6fd1a59528b957266c5f587d77e76c3391239db21525a783019a2c0e740cb4"} Oct 30 14:18:59.308444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.308405 5824 scope.go:122] "RemoveContainer" containerID="46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364" Oct 30 14:18:59.308581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.308570 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8" Oct 30 14:18:59.334201 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.334159 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6507c54e-2da9-4fb6-9014-14da28aa9ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6507c54e-2da9-4fb6-9014-14da28aa9ec8-plugins-config-volume\") pod \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\" (UID: \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\") " Oct 30 14:18:59.334334 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.334251 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6507c54e-2da9-4fb6-9014-14da28aa9ec8-kube-api-access-4vtvn\" (UniqueName: \"kubernetes.io/projected/6507c54e-2da9-4fb6-9014-14da28aa9ec8-kube-api-access-4vtvn\") pod \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\" (UID: \"6507c54e-2da9-4fb6-9014-14da28aa9ec8\") " Oct 30 14:18:59.334606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.334576 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6507c54e-2da9-4fb6-9014-14da28aa9ec8-plugins-config-volume" pod "6507c54e-2da9-4fb6-9014-14da28aa9ec8" (UID: "6507c54e-2da9-4fb6-9014-14da28aa9ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:59.335079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.335041 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6507c54e-2da9-4fb6-9014-14da28aa9ec8-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:59.336557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.336531 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6507c54e-2da9-4fb6-9014-14da28aa9ec8-kube-api-access-4vtvn" pod "6507c54e-2da9-4fb6-9014-14da28aa9ec8" (UID: "6507c54e-2da9-4fb6-9014-14da28aa9ec8"). InnerVolumeSpecName "kube-api-access-4vtvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:59.379965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.379938 5824 scope.go:122] "RemoveContainer" containerID="46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364" Oct 30 14:18:59.380348 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:18:59.380322 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364\": container with ID starting with 46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364 not found: ID does not exist" containerID="46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364" Oct 30 14:18:59.380435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.380353 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364"} err="failed to get container status \"46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364\": rpc error: code = NotFound desc = could not find container \"46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364\": container with ID starting with 46a2d014dfc979933ff54966794c739124154cb0a35f347aaf3b2c2bec6c4364 not found: ID does not exist" Oct 30 14:18:59.437022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.436983 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtvn\" (UniqueName: \"kubernetes.io/projected/6507c54e-2da9-4fb6-9014-14da28aa9ec8-kube-api-access-4vtvn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:59.524711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.524681 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:18:59.639595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.639569 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adec2d68-9246-43fe-9c98-22fbb8891fb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adec2d68-9246-43fe-9c98-22fbb8891fb6-plugins-config-volume\") pod \"adec2d68-9246-43fe-9c98-22fbb8891fb6\" (UID: \"adec2d68-9246-43fe-9c98-22fbb8891fb6\") " Oct 30 14:18:59.639735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.639659 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adec2d68-9246-43fe-9c98-22fbb8891fb6-kube-api-access-wmvtm\" (UniqueName: \"kubernetes.io/projected/adec2d68-9246-43fe-9c98-22fbb8891fb6-kube-api-access-wmvtm\") pod \"adec2d68-9246-43fe-9c98-22fbb8891fb6\" (UID: \"adec2d68-9246-43fe-9c98-22fbb8891fb6\") " Oct 30 14:18:59.639978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.639954 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adec2d68-9246-43fe-9c98-22fbb8891fb6-plugins-config-volume" pod "adec2d68-9246-43fe-9c98-22fbb8891fb6" (UID: "adec2d68-9246-43fe-9c98-22fbb8891fb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:18:59.640244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.640228 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adec2d68-9246-43fe-9c98-22fbb8891fb6-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:18:59.641614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.641592 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adec2d68-9246-43fe-9c98-22fbb8891fb6-kube-api-access-wmvtm" pod "adec2d68-9246-43fe-9c98-22fbb8891fb6" (UID: "adec2d68-9246-43fe-9c98-22fbb8891fb6"). InnerVolumeSpecName "kube-api-access-wmvtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:18:59.665532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.665504 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8"] Oct 30 14:18:59.676619 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.676590 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-01955278-7d4f98b5fd-ltzh8"] Oct 30 14:18:59.741370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:18:59.741348 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmvtm\" (UniqueName: \"kubernetes.io/projected/adec2d68-9246-43fe-9c98-22fbb8891fb6-kube-api-access-wmvtm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:19:00.350616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.350520 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6507c54e-2da9-4fb6-9014-14da28aa9ec8" path="/var/lib/kubelet/pods/6507c54e-2da9-4fb6-9014-14da28aa9ec8/volumes" Oct 30 14:19:00.356103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.356065 5824 generic.go:356] "Generic (PLEG): container finished" podID="adec2d68-9246-43fe-9c98-22fbb8891fb6" containerID="92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997" exitCode=0 Oct 30 14:19:00.356329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.356302 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" event={"ID":"adec2d68-9246-43fe-9c98-22fbb8891fb6","Type":"ContainerDied","Data":"92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997"} Oct 30 14:19:00.356406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.356368 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" event={"ID":"adec2d68-9246-43fe-9c98-22fbb8891fb6","Type":"ContainerDied","Data":"2d0ebe7d30af9221da58c09fa824dac60d3431ba7ba6b5ad84dca3fabfaeec6a"} Oct 30 14:19:00.356406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.356401 5824 scope.go:122] "RemoveContainer" containerID="92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997" Oct 30 14:19:00.356591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.356572 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq" Oct 30 14:19:00.383676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.383654 5824 scope.go:122] "RemoveContainer" containerID="92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997" Oct 30 14:19:00.383983 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:19:00.383965 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997\": container with ID starting with 92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997 not found: ID does not exist" containerID="92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997" Oct 30 14:19:00.384061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.383990 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997"} err="failed to get container status \"92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997\": rpc error: code = NotFound desc = could not find container \"92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997\": container with ID starting with 92bfcc8dcfaadcda76898fa787531f87c6e145a7e01b6fe30c96dfd09dcd6997 not found: ID does not exist" Oct 30 14:19:00.421000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.420974 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq"] Oct 30 14:19:00.434000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:00.433977 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-01955278-f55689567-97lbq"] Oct 30 14:19:02.320366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:02.320324 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adec2d68-9246-43fe-9c98-22fbb8891fb6" path="/var/lib/kubelet/pods/adec2d68-9246-43fe-9c98-22fbb8891fb6/volumes" Oct 30 14:19:03.641547 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:19:03.641510 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:19:19.035631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:19.035584 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9"] Oct 30 14:19:19.041432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:19.041395 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe66b9bc-rgtr9"] Oct 30 14:19:20.347213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:20.346241 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d918c2d-a799-485e-9b3d-92f2e73b68fe" path="/var/lib/kubelet/pods/8d918c2d-a799-485e-9b3d-92f2e73b68fe/volumes" Oct 30 14:19:24.322063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.322028 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b"] Oct 30 14:19:24.323180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323161 5824 cpu_manager.go:414] "containerMap: removing container" podUID="adec2d68-9246-43fe-9c98-22fbb8891fb6" containerName="payload-pre-processing" Oct 30 14:19:24.323251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323182 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adec2d68-9246-43fe-9c98-22fbb8891fb6" containerName="payload-pre-processing" Oct 30 14:19:24.323251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323219 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" Oct 30 14:19:24.323251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323224 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" Oct 30 14:19:24.323251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323236 5824 cpu_manager.go:414] "containerMap: removing container" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" Oct 30 14:19:24.323251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323241 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" Oct 30 14:19:24.323407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323271 5824 cpu_manager.go:414] "containerMap: removing container" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" Oct 30 14:19:24.323407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323275 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" Oct 30 14:19:24.323407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323291 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6507c54e-2da9-4fb6-9014-14da28aa9ec8" containerName="payload-processing" Oct 30 14:19:24.323407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323296 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6507c54e-2da9-4fb6-9014-14da28aa9ec8" containerName="payload-processing" Oct 30 14:19:24.323407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323303 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c" containerName="revoke-keys" Oct 30 14:19:24.323407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323308 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c" containerName="revoke-keys" Oct 30 14:19:24.323594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323424 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6507c54e-2da9-4fb6-9014-14da28aa9ec8" containerName="payload-processing" Oct 30 14:19:24.323594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323435 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c" containerName="revoke-keys" Oct 30 14:19:24.323594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323444 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="adec2d68-9246-43fe-9c98-22fbb8891fb6" containerName="payload-pre-processing" Oct 30 14:19:24.323594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323453 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c14891-26f7-43e9-8c8a-ff9e5fc3c735" containerName="istio-proxy" Oct 30 14:19:24.323594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323460 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a0a566-f79f-4fb4-b5be-9d2862d656a5" containerName="istio-proxy" Oct 30 14:19:24.323594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.323469 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c747c2-f178-4a1c-a666-634e8bc57fef" containerName="istio-proxy" Oct 30 14:19:24.327389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.327373 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:24.359650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.359623 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b"] Oct 30 14:19:24.367896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.367876 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1c4b82\"" Oct 30 14:19:24.368020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.367913 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1c4b82-dockercfg-9jnh2\"" Oct 30 14:19:24.368666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.368637 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950244f2-2998-4c57-8ec7-6a077ad9df7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b\" (UID: \"950244f2-2998-4c57-8ec7-6a077ad9df7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:24.368931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.368901 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sswps\" (UniqueName: \"kubernetes.io/projected/950244f2-2998-4c57-8ec7-6a077ad9df7e-kube-api-access-sswps\") pod \"payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b\" (UID: \"950244f2-2998-4c57-8ec7-6a077ad9df7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:24.472734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.472235 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950244f2-2998-4c57-8ec7-6a077ad9df7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b\" (UID: \"950244f2-2998-4c57-8ec7-6a077ad9df7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:24.473404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.473214 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950244f2-2998-4c57-8ec7-6a077ad9df7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b\" (UID: \"950244f2-2998-4c57-8ec7-6a077ad9df7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:24.473404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.473216 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswps\" (UniqueName: \"kubernetes.io/projected/950244f2-2998-4c57-8ec7-6a077ad9df7e-kube-api-access-sswps\") pod \"payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b\" (UID: \"950244f2-2998-4c57-8ec7-6a077ad9df7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:24.478479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.478451 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj"] Oct 30 14:19:24.489674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.489643 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:24.526551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.526517 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswps\" (UniqueName: \"kubernetes.io/projected/950244f2-2998-4c57-8ec7-6a077ad9df7e-kube-api-access-sswps\") pod \"payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b\" (UID: \"950244f2-2998-4c57-8ec7-6a077ad9df7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:24.560972 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.560932 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj"] Oct 30 14:19:24.586515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.586349 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj\" (UID: \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:24.586987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.586963 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knmp7\" (UniqueName: \"kubernetes.io/projected/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-kube-api-access-knmp7\") pod \"payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj\" (UID: \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:24.643638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.643611 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:24.690173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.689854 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knmp7\" (UniqueName: \"kubernetes.io/projected/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-kube-api-access-knmp7\") pod \"payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj\" (UID: \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:24.690173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.690080 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj\" (UID: \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:24.690820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.690795 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj\" (UID: \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:24.720919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.720864 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knmp7\" (UniqueName: \"kubernetes.io/projected/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-kube-api-access-knmp7\") pod \"payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj\" (UID: \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:24.823451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.823417 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:24.976965 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:19:24.976929 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950244f2_2998_4c57_8ec7_6a077ad9df7e.slice/crio-8533a77d4b56f73974cc7ea528247655fff39e86d10d697e4b36caa0af382e83 WatchSource:0}: Error finding container 8533a77d4b56f73974cc7ea528247655fff39e86d10d697e4b36caa0af382e83: Status 404 returned error can't find the container with id 8533a77d4b56f73974cc7ea528247655fff39e86d10d697e4b36caa0af382e83 Oct 30 14:19:24.977355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:24.977333 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b"] Oct 30 14:19:25.134846 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:19:25.134808 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-67f4d83b6c7314094b6e6cc755733f54bad0540b03dd25f707f2e3b84adb65a6 WatchSource:0}: Error finding container 67f4d83b6c7314094b6e6cc755733f54bad0540b03dd25f707f2e3b84adb65a6: Status 404 returned error can't find the container with id 67f4d83b6c7314094b6e6cc755733f54bad0540b03dd25f707f2e3b84adb65a6 Oct 30 14:19:25.137226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:25.137200 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj"] Oct 30 14:19:25.659265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:25.659232 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" event={"ID":"bd3ed73e-5532-4eea-8e95-cd2c95c3c235","Type":"ContainerStarted","Data":"cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907"} Oct 30 14:19:25.659742 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:25.659282 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" event={"ID":"bd3ed73e-5532-4eea-8e95-cd2c95c3c235","Type":"ContainerStarted","Data":"67f4d83b6c7314094b6e6cc755733f54bad0540b03dd25f707f2e3b84adb65a6"} Oct 30 14:19:25.659742 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:25.659309 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:25.661380 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:25.661337 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" event={"ID":"950244f2-2998-4c57-8ec7-6a077ad9df7e","Type":"ContainerStarted","Data":"b0770d278c8e2cbba0ecabe1a8b2dd243ff3378da12cc72c4b2e39d58624644f"} Oct 30 14:19:25.661564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:25.661390 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" event={"ID":"950244f2-2998-4c57-8ec7-6a077ad9df7e","Type":"ContainerStarted","Data":"8533a77d4b56f73974cc7ea528247655fff39e86d10d697e4b36caa0af382e83"} Oct 30 14:19:25.661643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:25.661572 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:25.715755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:25.715686 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" podStartSLOduration=1.715669452 podStartE2EDuration="1.715669452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:25.714883995 +0000 UTC m=+1795.515389456" watchObservedRunningTime="2026-10-30 14:19:25.715669452 +0000 UTC m=+1795.516174895" Oct 30 14:19:26.310297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.310239 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" podStartSLOduration=2.310224022 podStartE2EDuration="2.310224022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:25.770722884 +0000 UTC m=+1795.571228329" watchObservedRunningTime="2026-10-30 14:19:26.310224022 +0000 UTC m=+1796.110729465" Oct 30 14:19:26.319664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.319636 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b"] Oct 30 14:19:26.330634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.330605 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:26.337913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.337887 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4"] Oct 30 14:19:26.347970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.342934 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:26.349587 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.349383 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b"] Oct 30 14:19:26.357253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.357092 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2a2835\"" Oct 30 14:19:26.364665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.364643 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4"] Oct 30 14:19:26.416697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.416664 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ksxv\" (UniqueName: \"kubernetes.io/projected/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-kube-api-access-4ksxv\") pod \"payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4\" (UID: \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:26.416859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.416741 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2crk\" (UniqueName: \"kubernetes.io/projected/3edfffcf-6f39-4b5d-b388-0a189dc55d95-kube-api-access-x2crk\") pod \"payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b\" (UID: \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:26.416944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.416886 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4\" (UID: \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:26.417119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.417101 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edfffcf-6f39-4b5d-b388-0a189dc55d95-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b\" (UID: \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:26.530210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.529221 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4\" (UID: \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:26.530210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.529501 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edfffcf-6f39-4b5d-b388-0a189dc55d95-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b\" (UID: \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:26.530210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.529625 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ksxv\" (UniqueName: \"kubernetes.io/projected/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-kube-api-access-4ksxv\") pod \"payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4\" (UID: \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:26.530210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.529690 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2crk\" (UniqueName: \"kubernetes.io/projected/3edfffcf-6f39-4b5d-b388-0a189dc55d95-kube-api-access-x2crk\") pod \"payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b\" (UID: \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:26.534230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.530937 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4\" (UID: \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:26.534230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.531702 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edfffcf-6f39-4b5d-b388-0a189dc55d95-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b\" (UID: \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:26.601746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.594759 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2crk\" (UniqueName: \"kubernetes.io/projected/3edfffcf-6f39-4b5d-b388-0a189dc55d95-kube-api-access-x2crk\") pod \"payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b\" (UID: \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:26.625208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.615154 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ksxv\" (UniqueName: \"kubernetes.io/projected/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-kube-api-access-4ksxv\") pod \"payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4\" (UID: \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:26.652215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.651137 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:26.671236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:26.668705 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:27.015292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.015267 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b"] Oct 30 14:19:27.015477 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:19:27.015453 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edfffcf_6f39_4b5d_b388_0a189dc55d95.slice/crio-c6822e1e79a5ed6512bd03e7a3f37778024771356ebecbc07350b224b2e8c349 WatchSource:0}: Error finding container c6822e1e79a5ed6512bd03e7a3f37778024771356ebecbc07350b224b2e8c349: Status 404 returned error can't find the container with id c6822e1e79a5ed6512bd03e7a3f37778024771356ebecbc07350b224b2e8c349 Oct 30 14:19:27.018307 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:19:27.018284 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027d8eba_cc3e_4f14_8c89_767e1ef25ea7.slice/crio-3e504b2e6b2d3e6899e43ab9d6ebff703d1071859a2f2b06fc901025282c1b2c WatchSource:0}: Error finding container 3e504b2e6b2d3e6899e43ab9d6ebff703d1071859a2f2b06fc901025282c1b2c: Status 404 returned error can't find the container with id 3e504b2e6b2d3e6899e43ab9d6ebff703d1071859a2f2b06fc901025282c1b2c Oct 30 14:19:27.021273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.020148 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4"] Oct 30 14:19:27.686604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.686568 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" event={"ID":"3edfffcf-6f39-4b5d-b388-0a189dc55d95","Type":"ContainerStarted","Data":"7234dacb48fa18d963ae89acb5e8fd3ea419eb180c472294f01d8c0a921a97dc"} Oct 30 14:19:27.687059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.686635 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" event={"ID":"3edfffcf-6f39-4b5d-b388-0a189dc55d95","Type":"ContainerStarted","Data":"c6822e1e79a5ed6512bd03e7a3f37778024771356ebecbc07350b224b2e8c349"} Oct 30 14:19:27.687059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.686664 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:27.688643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.688615 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" event={"ID":"027d8eba-cc3e-4f14-8c89-767e1ef25ea7","Type":"ContainerStarted","Data":"cfe01f4d7dcaedc331f275686f26a9665764438c67619dc582f8d5a8e9d733ab"} Oct 30 14:19:27.688766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.688660 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" event={"ID":"027d8eba-cc3e-4f14-8c89-767e1ef25ea7","Type":"ContainerStarted","Data":"3e504b2e6b2d3e6899e43ab9d6ebff703d1071859a2f2b06fc901025282c1b2c"} Oct 30 14:19:27.688766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.688686 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:27.741281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.741219 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" podStartSLOduration=1.7411803030000002 podStartE2EDuration="1.741180303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:27.740588804 +0000 UTC m=+1797.541094252" watchObservedRunningTime="2026-10-30 14:19:27.741180303 +0000 UTC m=+1797.541685752" Oct 30 14:19:27.789512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:27.789459 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" podStartSLOduration=1.789445698 podStartE2EDuration="1.789445698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:27.786507355 +0000 UTC m=+1797.587012798" watchObservedRunningTime="2026-10-30 14:19:27.789445698 +0000 UTC m=+1797.589951142" Oct 30 14:19:31.170528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:31.170496 5824 scope.go:122] "RemoveContainer" containerID="c1ade2f123003403ac215a8bbb4c53e76922c12d8f714c6c78f694ce9a890ba0" Oct 30 14:19:33.647181 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:19:33.647145 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:19:36.677669 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:36.677640 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:19:36.678061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:36.677724 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:19:38.708142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:38.708105 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:19:38.711055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:38.711020 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:19:52.802339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.802255 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx"] Oct 30 14:19:52.813713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.813687 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:52.822254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.822229 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx"] Oct 30 14:19:52.831772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.831742 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5a9d7b-dockercfg-9hrh8\"" Oct 30 14:19:52.844003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.843975 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5a9d7b\"" Oct 30 14:19:52.872668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.872643 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65da2b41-a7a1-47b2-89ce-424fba8b82a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx\" (UID: \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:52.873069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.873031 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5g8f\" (UniqueName: \"kubernetes.io/projected/65da2b41-a7a1-47b2-89ce-424fba8b82a5-kube-api-access-x5g8f\") pod \"payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx\" (UID: \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:52.996212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.982523 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65da2b41-a7a1-47b2-89ce-424fba8b82a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx\" (UID: \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:52.996212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.982720 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5g8f\" (UniqueName: \"kubernetes.io/projected/65da2b41-a7a1-47b2-89ce-424fba8b82a5-kube-api-access-x5g8f\") pod \"payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx\" (UID: \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:52.996212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:52.984256 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65da2b41-a7a1-47b2-89ce-424fba8b82a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx\" (UID: \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:53.062208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:53.062113 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5g8f\" (UniqueName: \"kubernetes.io/projected/65da2b41-a7a1-47b2-89ce-424fba8b82a5-kube-api-access-x5g8f\") pod \"payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx\" (UID: \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:53.133035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:53.132801 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:53.458233 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:19:53.458054 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65da2b41_a7a1_47b2_89ce_424fba8b82a5.slice/crio-55d46015db325e7290966e1e7c72b40e5f05ec8190dc1cd85c583d8fcad26aa1 WatchSource:0}: Error finding container 55d46015db325e7290966e1e7c72b40e5f05ec8190dc1cd85c583d8fcad26aa1: Status 404 returned error can't find the container with id 55d46015db325e7290966e1e7c72b40e5f05ec8190dc1cd85c583d8fcad26aa1 Oct 30 14:19:53.465871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:53.465844 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx"] Oct 30 14:19:53.952609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:53.952570 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" event={"ID":"65da2b41-a7a1-47b2-89ce-424fba8b82a5","Type":"ContainerStarted","Data":"0a1f1f5a7d64419a86e8c53ce93fce0f437dc89dc61ccbced62cd1e2445a2207"} Oct 30 14:19:53.953018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:53.952629 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" event={"ID":"65da2b41-a7a1-47b2-89ce-424fba8b82a5","Type":"ContainerStarted","Data":"55d46015db325e7290966e1e7c72b40e5f05ec8190dc1cd85c583d8fcad26aa1"} Oct 30 14:19:53.953018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:53.952668 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:19:54.002821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:19:54.002773 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" podStartSLOduration=2.002758624 podStartE2EDuration="2.002758624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:19:53.998152129 +0000 UTC m=+1823.798657568" watchObservedRunningTime="2026-10-30 14:19:54.002758624 +0000 UTC m=+1823.803264068" Oct 30 14:20:03.643451 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:20:03.643412 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:20:04.965310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:04.965282 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:20:15.379570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:15.379536 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f"] Oct 30 14:20:15.417953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:15.417916 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn"] Oct 30 14:20:20.380747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.380696 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" podUID="578ef5f4-d95a-4384-965e-dc8561763917" containerName="payload-processing" containerID="cri-o://d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f" gracePeriod=25 Oct 30 14:20:20.418964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.418930 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" podUID="84afee28-cab0-479b-9867-d2605054f28a" containerName="payload-pre-processing" containerID="cri-o://11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41" gracePeriod=25 Oct 30 14:20:20.716944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.716914 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:20:20.761307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.761284 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:20:20.836884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.836852 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84afee28-cab0-479b-9867-d2605054f28a-kube-api-access-qmxww\" (UniqueName: \"kubernetes.io/projected/84afee28-cab0-479b-9867-d2605054f28a-kube-api-access-qmxww\") pod \"84afee28-cab0-479b-9867-d2605054f28a\" (UID: \"84afee28-cab0-479b-9867-d2605054f28a\") " Oct 30 14:20:20.837055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.837012 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84afee28-cab0-479b-9867-d2605054f28a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afee28-cab0-479b-9867-d2605054f28a-plugins-config-volume\") pod \"84afee28-cab0-479b-9867-d2605054f28a\" (UID: \"84afee28-cab0-479b-9867-d2605054f28a\") " Oct 30 14:20:20.837055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.837049 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578ef5f4-d95a-4384-965e-dc8561763917-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578ef5f4-d95a-4384-965e-dc8561763917-plugins-config-volume\") pod \"578ef5f4-d95a-4384-965e-dc8561763917\" (UID: \"578ef5f4-d95a-4384-965e-dc8561763917\") " Oct 30 14:20:20.838118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.837267 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578ef5f4-d95a-4384-965e-dc8561763917-kube-api-access-x46lz\" (UniqueName: \"kubernetes.io/projected/578ef5f4-d95a-4384-965e-dc8561763917-kube-api-access-x46lz\") pod \"578ef5f4-d95a-4384-965e-dc8561763917\" (UID: \"578ef5f4-d95a-4384-965e-dc8561763917\") " Oct 30 14:20:20.838118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.837504 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578ef5f4-d95a-4384-965e-dc8561763917-plugins-config-volume" pod "578ef5f4-d95a-4384-965e-dc8561763917" (UID: "578ef5f4-d95a-4384-965e-dc8561763917"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:20:20.838118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.837919 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84afee28-cab0-479b-9867-d2605054f28a-plugins-config-volume" pod "84afee28-cab0-479b-9867-d2605054f28a" (UID: "84afee28-cab0-479b-9867-d2605054f28a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:20:20.838118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.838070 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afee28-cab0-479b-9867-d2605054f28a-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:20:20.838118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.838095 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578ef5f4-d95a-4384-965e-dc8561763917-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:20:20.839414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.839390 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84afee28-cab0-479b-9867-d2605054f28a-kube-api-access-qmxww" pod "84afee28-cab0-479b-9867-d2605054f28a" (UID: "84afee28-cab0-479b-9867-d2605054f28a"). InnerVolumeSpecName "kube-api-access-qmxww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:20.839870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.839851 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578ef5f4-d95a-4384-965e-dc8561763917-kube-api-access-x46lz" pod "578ef5f4-d95a-4384-965e-dc8561763917" (UID: "578ef5f4-d95a-4384-965e-dc8561763917"). InnerVolumeSpecName "kube-api-access-x46lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:20.939038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.939008 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46lz\" (UniqueName: \"kubernetes.io/projected/578ef5f4-d95a-4384-965e-dc8561763917-kube-api-access-x46lz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:20:20.939038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:20.939039 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxww\" (UniqueName: \"kubernetes.io/projected/84afee28-cab0-479b-9867-d2605054f28a-kube-api-access-qmxww\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:20:21.265837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.265759 5824 generic.go:356] "Generic (PLEG): container finished" podID="84afee28-cab0-479b-9867-d2605054f28a" containerID="11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41" exitCode=0 Oct 30 14:20:21.265837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.265812 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" Oct 30 14:20:21.266014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.265844 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" event={"ID":"84afee28-cab0-479b-9867-d2605054f28a","Type":"ContainerDied","Data":"11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41"} Oct 30 14:20:21.266014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.265901 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn" event={"ID":"84afee28-cab0-479b-9867-d2605054f28a","Type":"ContainerDied","Data":"f5e6bc0c2904c90376ba2a82a2b1c09097422ec20260d106243721abf124450a"} Oct 30 14:20:21.266014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.265930 5824 scope.go:122] "RemoveContainer" containerID="11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41" Oct 30 14:20:21.268284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.268244 5824 generic.go:356] "Generic (PLEG): container finished" podID="578ef5f4-d95a-4384-965e-dc8561763917" containerID="d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f" exitCode=0 Oct 30 14:20:21.268418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.268321 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" Oct 30 14:20:21.268418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.268402 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" event={"ID":"578ef5f4-d95a-4384-965e-dc8561763917","Type":"ContainerDied","Data":"d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f"} Oct 30 14:20:21.268532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.268446 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f" event={"ID":"578ef5f4-d95a-4384-965e-dc8561763917","Type":"ContainerDied","Data":"899dcc1b302e1a2ce2ada54c9ef068329a47b8c629a4117869ee9bdc4361a39d"} Oct 30 14:20:21.278996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.278762 5824 scope.go:122] "RemoveContainer" containerID="11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41" Oct 30 14:20:21.279123 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:20:21.279092 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41\": container with ID starting with 11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41 not found: ID does not exist" containerID="11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41" Oct 30 14:20:21.279274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.279131 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41"} err="failed to get container status \"11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41\": rpc error: code = NotFound desc = could not find container \"11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41\": container with ID starting with 11fd8b72ca9a57d0a8f0e2317cae0ff865a2d2815c755af01d2b3fe1be770b41 not found: ID does not exist" Oct 30 14:20:21.279274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.279148 5824 scope.go:122] "RemoveContainer" containerID="d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f" Oct 30 14:20:21.288954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.288934 5824 scope.go:122] "RemoveContainer" containerID="d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f" Oct 30 14:20:21.289398 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:20:21.289379 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f\": container with ID starting with d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f not found: ID does not exist" containerID="d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f" Oct 30 14:20:21.289483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.289404 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f"} err="failed to get container status \"d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f\": rpc error: code = NotFound desc = could not find container \"d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f\": container with ID starting with d2d27129f58b05653b1b5223c84faf855f056991c5690535613febb0e4f98e0f not found: ID does not exist" Oct 30 14:20:21.333546 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.333515 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn"] Oct 30 14:20:21.346561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.346536 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d727dc-db644f5ff-cg8nn"] Oct 30 14:20:21.373157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.373117 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f"] Oct 30 14:20:21.380697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:21.380670 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d727dc-bc4fc7c88-6sw2f"] Oct 30 14:20:22.320059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:22.320026 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578ef5f4-d95a-4384-965e-dc8561763917" path="/var/lib/kubelet/pods/578ef5f4-d95a-4384-965e-dc8561763917/volumes" Oct 30 14:20:22.320542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:22.320523 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84afee28-cab0-479b-9867-d2605054f28a" path="/var/lib/kubelet/pods/84afee28-cab0-479b-9867-d2605054f28a/volumes" Oct 30 14:20:31.223698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:31.223666 5824 scope.go:122] "RemoveContainer" containerID="0cb69832aecc25d224a86198075e1f20549581168c802585f3d8737a4f5068e4" Oct 30 14:20:31.234109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:31.234089 5824 scope.go:122] "RemoveContainer" containerID="ec027e5b69a387d563a44d27606a97fdb3656b50d615c536ad6b4056ce2bc023" Oct 30 14:20:31.668897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:31.668824 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78"] Oct 30 14:20:33.641717 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:20:33.641682 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:20:36.669547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:36.669494 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" podUID="b0edc37e-f5bf-48e8-a088-ee66527dd569" containerName="payload-processing" containerID="cri-o://a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac" gracePeriod=25 Oct 30 14:20:36.976489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:36.976463 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:20:37.038423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.038393 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0edc37e-f5bf-48e8-a088-ee66527dd569-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0edc37e-f5bf-48e8-a088-ee66527dd569-plugins-config-volume\") pod \"b0edc37e-f5bf-48e8-a088-ee66527dd569\" (UID: \"b0edc37e-f5bf-48e8-a088-ee66527dd569\") " Oct 30 14:20:37.038646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.038632 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0edc37e-f5bf-48e8-a088-ee66527dd569-kube-api-access-9wdj4\" (UniqueName: \"kubernetes.io/projected/b0edc37e-f5bf-48e8-a088-ee66527dd569-kube-api-access-9wdj4\") pod \"b0edc37e-f5bf-48e8-a088-ee66527dd569\" (UID: \"b0edc37e-f5bf-48e8-a088-ee66527dd569\") " Oct 30 14:20:37.038829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.038800 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0edc37e-f5bf-48e8-a088-ee66527dd569-plugins-config-volume" pod "b0edc37e-f5bf-48e8-a088-ee66527dd569" (UID: "b0edc37e-f5bf-48e8-a088-ee66527dd569"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:20:37.039311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.039288 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0edc37e-f5bf-48e8-a088-ee66527dd569-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:20:37.041718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.041671 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0edc37e-f5bf-48e8-a088-ee66527dd569-kube-api-access-9wdj4" pod "b0edc37e-f5bf-48e8-a088-ee66527dd569" (UID: "b0edc37e-f5bf-48e8-a088-ee66527dd569"). InnerVolumeSpecName "kube-api-access-9wdj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:20:37.140251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.140223 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wdj4\" (UniqueName: \"kubernetes.io/projected/b0edc37e-f5bf-48e8-a088-ee66527dd569-kube-api-access-9wdj4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:20:37.463262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.463230 5824 generic.go:356] "Generic (PLEG): container finished" podID="b0edc37e-f5bf-48e8-a088-ee66527dd569" containerID="a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac" exitCode=0 Oct 30 14:20:37.463262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.463256 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" Oct 30 14:20:37.463461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.463302 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" event={"ID":"b0edc37e-f5bf-48e8-a088-ee66527dd569","Type":"ContainerDied","Data":"a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac"} Oct 30 14:20:37.463461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.463354 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78" event={"ID":"b0edc37e-f5bf-48e8-a088-ee66527dd569","Type":"ContainerDied","Data":"b5132c91525cf2071d7342b381117904cd943308f7e6f9acf0cde1b5ea04256b"} Oct 30 14:20:37.463461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.463384 5824 scope.go:122] "RemoveContainer" containerID="a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac" Oct 30 14:20:37.475383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.475364 5824 scope.go:122] "RemoveContainer" containerID="a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac" Oct 30 14:20:37.475663 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:20:37.475632 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac\": container with ID starting with a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac not found: ID does not exist" containerID="a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac" Oct 30 14:20:37.475775 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.475668 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac"} err="failed to get container status \"a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac\": rpc error: code = NotFound desc = could not find container \"a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac\": container with ID starting with a7d9a871b49825d35b9e3b674f1d30ead2f12a7ea6906fd7a4fde44f221368ac not found: ID does not exist" Oct 30 14:20:37.522856 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.522830 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78"] Oct 30 14:20:37.538970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:37.538944 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-972fee-546d4b6686-svw78"] Oct 30 14:20:38.328519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:38.328480 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0edc37e-f5bf-48e8-a088-ee66527dd569" path="/var/lib/kubelet/pods/b0edc37e-f5bf-48e8-a088-ee66527dd569/volumes" Oct 30 14:20:57.675490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:57.675458 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk"] Oct 30 14:20:57.675911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:20:57.675671 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" containerID="cri-o://a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d" gracePeriod=30 Oct 30 14:21:02.968879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:02.968851 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:21:03.035131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035096 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-envoy\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.035343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035212 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-kube-api-access-zzkf7\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-kube-api-access-zzkf7\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.035343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035267 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7d6e232-fd91-466c-befa-f8db13723638-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d6e232-fd91-466c-befa-f8db13723638-istio-podinfo\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.035343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035341 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-credential-socket\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.035516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035387 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-data\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.035565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035521 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d6e232-fd91-466c-befa-f8db13723638-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6e232-fd91-466c-befa-f8db13723638-istiod-ca-cert\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.035727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035687 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-credential-socket" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:03.035727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035717 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-certs\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.035895 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035817 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-data" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:03.035895 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035828 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-socket\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.036008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035949 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-istio-token\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-istio-token\") pod \"b7d6e232-fd91-466c-befa-f8db13723638\" (UID: \"b7d6e232-fd91-466c-befa-f8db13723638\") " Oct 30 14:21:03.036008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.035993 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-socket" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:03.036100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.036042 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d6e232-fd91-466c-befa-f8db13723638-istiod-ca-cert" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:03.036214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.036176 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-certs" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:03.036847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.036813 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.036847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.036845 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.037016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.036862 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6e232-fd91-466c-befa-f8db13723638-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.037016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.036876 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.037016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.036889 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.038098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.038066 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-kube-api-access-zzkf7" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "kube-api-access-zzkf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:03.038098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.038081 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7d6e232-fd91-466c-befa-f8db13723638-istio-podinfo" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:21:03.038328 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.038283 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-istio-token" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:03.042363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.042342 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-envoy" pod "b7d6e232-fd91-466c-befa-f8db13723638" (UID: "b7d6e232-fd91-466c-befa-f8db13723638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:03.138840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.138812 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzkf7\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-kube-api-access-zzkf7\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.138840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.138839 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d6e232-fd91-466c-befa-f8db13723638-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.138840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.138848 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d6e232-fd91-466c-befa-f8db13723638-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.139049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.138856 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d6e232-fd91-466c-befa-f8db13723638-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:03.643636 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:21:03.643603 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:21:03.716438 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.716406 5824 generic.go:356] "Generic (PLEG): container finished" podID="b7d6e232-fd91-466c-befa-f8db13723638" containerID="a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d" exitCode=0 Oct 30 14:21:03.716589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.716454 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" Oct 30 14:21:03.716589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.716498 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" event={"ID":"b7d6e232-fd91-466c-befa-f8db13723638","Type":"ContainerDied","Data":"a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d"} Oct 30 14:21:03.716589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.716563 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk" event={"ID":"b7d6e232-fd91-466c-befa-f8db13723638","Type":"ContainerDied","Data":"41c196d57331f3afc31c1efd39bb0bd7b07f7d685d558e1156a6f522f944239e"} Oct 30 14:21:03.716759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.716593 5824 scope.go:122] "RemoveContainer" containerID="a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d" Oct 30 14:21:03.728145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.728126 5824 scope.go:122] "RemoveContainer" containerID="a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d" Oct 30 14:21:03.728428 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:21:03.728408 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d\": container with ID starting with a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d not found: ID does not exist" containerID="a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d" Oct 30 14:21:03.728508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.728434 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d"} err="failed to get container status \"a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d\": rpc error: code = NotFound desc = could not find container \"a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d\": container with ID starting with a4f5a200c4e71ece90c0c4ff3a57c33ff9222380da3a2a8161ca06d59a4e337d not found: ID does not exist" Oct 30 14:21:03.791026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.791007 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk"] Oct 30 14:21:03.804235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:03.804212 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-7cjsk"] Oct 30 14:21:04.319207 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:04.319169 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d6e232-fd91-466c-befa-f8db13723638" path="/var/lib/kubelet/pods/b7d6e232-fd91-466c-befa-f8db13723638/volumes" Oct 30 14:21:10.229004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.228972 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z"] Oct 30 14:21:10.229751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229736 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" Oct 30 14:21:10.229796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229752 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" Oct 30 14:21:10.229796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229776 5824 cpu_manager.go:414] "containerMap: removing container" podUID="578ef5f4-d95a-4384-965e-dc8561763917" containerName="payload-processing" Oct 30 14:21:10.229796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229781 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578ef5f4-d95a-4384-965e-dc8561763917" containerName="payload-processing" Oct 30 14:21:10.229886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229803 5824 cpu_manager.go:414] "containerMap: removing container" podUID="84afee28-cab0-479b-9867-d2605054f28a" containerName="payload-pre-processing" Oct 30 14:21:10.229886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229809 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84afee28-cab0-479b-9867-d2605054f28a" containerName="payload-pre-processing" Oct 30 14:21:10.229886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229822 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b0edc37e-f5bf-48e8-a088-ee66527dd569" containerName="payload-processing" Oct 30 14:21:10.229886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229827 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0edc37e-f5bf-48e8-a088-ee66527dd569" containerName="payload-processing" Oct 30 14:21:10.230004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229954 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d6e232-fd91-466c-befa-f8db13723638" containerName="istio-proxy" Oct 30 14:21:10.230004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229964 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="578ef5f4-d95a-4384-965e-dc8561763917" containerName="payload-processing" Oct 30 14:21:10.230004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229975 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="84afee28-cab0-479b-9867-d2605054f28a" containerName="payload-pre-processing" Oct 30 14:21:10.230004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.229983 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0edc37e-f5bf-48e8-a088-ee66527dd569" containerName="payload-processing" Oct 30 14:21:10.235327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.235308 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.238691 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.238663 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.238851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.238815 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.238971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.238897 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6860cde2-f7a8-42e0-b3c9-c02000870156-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.239085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.239036 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.239226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.239092 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.239226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.239218 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.239447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.239378 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbcr\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-kube-api-access-zgbcr\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.240019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.239663 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.240019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.239771 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.258656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.258628 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z"] Oct 30 14:21:10.340083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.340060 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2a2835-openshift-default-dockercfg-s5hmr\"" Oct 30 14:21:10.340600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.340582 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.340700 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.340686 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.340744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.340712 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6860cde2-f7a8-42e0-b3c9-c02000870156-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.340794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.340761 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.340849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.340816 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.340933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.340916 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.341009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.340988 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbcr\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-kube-api-access-zgbcr\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.341071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.341015 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.341071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.341035 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.341169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.341082 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.341285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.341082 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.341366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.341296 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.341516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.341491 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6860cde2-f7a8-42e0-b3c9-c02000870156-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.341681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.341546 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.344933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.344911 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.345017 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.344988 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.366567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.366542 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbcr\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-kube-api-access-zgbcr\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.366689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.366669 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.411647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.411622 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf"] Oct 30 14:21:10.446508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.446487 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c"] Oct 30 14:21:10.561693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.561607 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:10.889433 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:21:10.889404 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6860cde2_f7a8_42e0_b3c9_c02000870156.slice/crio-548c7fd93948de35dfe87de0b8a55d71cde3306ea48e0d5daf5e2918e22f8039 WatchSource:0}: Error finding container 548c7fd93948de35dfe87de0b8a55d71cde3306ea48e0d5daf5e2918e22f8039: Status 404 returned error can't find the container with id 548c7fd93948de35dfe87de0b8a55d71cde3306ea48e0d5daf5e2918e22f8039 Oct 30 14:21:10.889659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.889638 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z"] Oct 30 14:21:10.892251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.892214 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:10.892375 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.892290 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:10.892375 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:10.892335 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:11.299071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:11.299040 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq"] Oct 30 14:21:11.797600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:11.797557 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" event={"ID":"6860cde2-f7a8-42e0-b3c9-c02000870156","Type":"ContainerStarted","Data":"13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193"} Oct 30 14:21:11.797761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:11.797624 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" event={"ID":"6860cde2-f7a8-42e0-b3c9-c02000870156","Type":"ContainerStarted","Data":"548c7fd93948de35dfe87de0b8a55d71cde3306ea48e0d5daf5e2918e22f8039"} Oct 30 14:21:11.870347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:11.865072 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" podStartSLOduration=1.86505136 podStartE2EDuration="1.86505136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:11.862856616 +0000 UTC m=+1901.663362060" watchObservedRunningTime="2026-10-30 14:21:11.86505136 +0000 UTC m=+1901.665556804" Oct 30 14:21:12.010818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:12.010785 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s"] Oct 30 14:21:12.010999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:12.010984 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" containerID="cri-o://5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a" gracePeriod=30 Oct 30 14:21:12.566688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:12.566515 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:12.567980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:12.567944 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:21:12.568111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:12.568006 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 14:21:13.566547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:13.566509 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:21:13.566725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:13.566571 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 14:21:14.566521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:14.566486 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:21:14.566734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:14.566547 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 14:21:15.412573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.412509 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" podUID="2a1fb9bc-41d4-46e2-b760-4c370821dd4e" containerName="payload-processing" containerID="cri-o://c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88" gracePeriod=25 Oct 30 14:21:15.453325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.453261 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" podUID="b33d70fd-8ff6-4d54-8b1d-0dae28c945d5" containerName="payload-pre-processing" containerID="cri-o://ac3152cb7349407cb85b0b5fdab3a1ecb1bb6f34e1d05619f5fcad1038257361" gracePeriod=25 Oct 30 14:21:15.568224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.567526 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:21:15.568224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.567587 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 14:21:15.840260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.839929 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:21:15.840912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.840883 5824 generic.go:356] "Generic (PLEG): container finished" podID="b33d70fd-8ff6-4d54-8b1d-0dae28c945d5" containerID="ac3152cb7349407cb85b0b5fdab3a1ecb1bb6f34e1d05619f5fcad1038257361" exitCode=0 Oct 30 14:21:15.841049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.840953 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" event={"ID":"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5","Type":"ContainerDied","Data":"ac3152cb7349407cb85b0b5fdab3a1ecb1bb6f34e1d05619f5fcad1038257361"} Oct 30 14:21:15.898940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.898894 5824 generic.go:356] "Generic (PLEG): container finished" podID="2a1fb9bc-41d4-46e2-b760-4c370821dd4e" containerID="c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88" exitCode=0 Oct 30 14:21:15.899139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.899025 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" event={"ID":"2a1fb9bc-41d4-46e2-b760-4c370821dd4e","Type":"ContainerDied","Data":"c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88"} Oct 30 14:21:15.899139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.899083 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" event={"ID":"2a1fb9bc-41d4-46e2-b760-4c370821dd4e","Type":"ContainerDied","Data":"6b56f57f851e778167a43f3ded2f956041ec509b15ef259aa5491384e288193f"} Oct 30 14:21:15.899139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.899118 5824 scope.go:122] "RemoveContainer" containerID="c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88" Oct 30 14:21:15.899416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.899381 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf" Oct 30 14:21:15.918614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.918594 5824 scope.go:122] "RemoveContainer" containerID="c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88" Oct 30 14:21:15.918882 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:21:15.918861 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88\": container with ID starting with c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88 not found: ID does not exist" containerID="c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88" Oct 30 14:21:15.918981 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.918893 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88"} err="failed to get container status \"c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88\": rpc error: code = NotFound desc = could not find container \"c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88\": container with ID starting with c1cef8473bd5935993fb7243f8a6a3f68ff263cd5f39d4040c50bcb28a5f3e88 not found: ID does not exist" Oct 30 14:21:15.984004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.983980 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:21:15.989932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.989910 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-plugins-config-volume\") pod \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\" (UID: \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\") " Oct 30 14:21:15.990134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.990114 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-kube-api-access-nk76f\" (UniqueName: \"kubernetes.io/projected/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-kube-api-access-nk76f\") pod \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\" (UID: \"2a1fb9bc-41d4-46e2-b760-4c370821dd4e\") " Oct 30 14:21:15.990360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.990336 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-plugins-config-volume" pod "2a1fb9bc-41d4-46e2-b760-4c370821dd4e" (UID: "2a1fb9bc-41d4-46e2-b760-4c370821dd4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:15.990901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.990875 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:15.992446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:15.992426 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-kube-api-access-nk76f" pod "2a1fb9bc-41d4-46e2-b760-4c370821dd4e" (UID: "2a1fb9bc-41d4-46e2-b760-4c370821dd4e"). InnerVolumeSpecName "kube-api-access-nk76f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:16.092617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.092542 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-plugins-config-volume\") pod \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\" (UID: \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\") " Oct 30 14:21:16.092751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.092665 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-kube-api-access-p9lg8\" (UniqueName: \"kubernetes.io/projected/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-kube-api-access-p9lg8\") pod \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\" (UID: \"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5\") " Oct 30 14:21:16.092982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.092948 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-plugins-config-volume" pod "b33d70fd-8ff6-4d54-8b1d-0dae28c945d5" (UID: "b33d70fd-8ff6-4d54-8b1d-0dae28c945d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:16.093310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.093291 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk76f\" (UniqueName: \"kubernetes.io/projected/2a1fb9bc-41d4-46e2-b760-4c370821dd4e-kube-api-access-nk76f\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:16.093310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.093310 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:16.094923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.094893 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-kube-api-access-p9lg8" pod "b33d70fd-8ff6-4d54-8b1d-0dae28c945d5" (UID: "b33d70fd-8ff6-4d54-8b1d-0dae28c945d5"). InnerVolumeSpecName "kube-api-access-p9lg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:16.124609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.124583 5824 patch_prober.go:29] interesting pod/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:21:16.124731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.124637 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:21:16.194724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.194702 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lg8\" (UniqueName: \"kubernetes.io/projected/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5-kube-api-access-p9lg8\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:16.271228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.271172 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf"] Oct 30 14:21:16.274686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.274662 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d74808-5ff59bf748-5cpkf"] Oct 30 14:21:16.299626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.299594 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" podUID="c7b624cf-d772-4725-bad1-74d15fe3626f" containerName="payload-processing" containerID="cri-o://fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc" gracePeriod=25 Oct 30 14:21:16.318713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.318692 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1fb9bc-41d4-46e2-b760-4c370821dd4e" path="/var/lib/kubelet/pods/2a1fb9bc-41d4-46e2-b760-4c370821dd4e/volumes" Oct 30 14:21:16.573928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.573890 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 14:21:16.574447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.573953 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 14:21:16.621071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.621013 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:21:16.701485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.701441 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b624cf-d772-4725-bad1-74d15fe3626f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b624cf-d772-4725-bad1-74d15fe3626f-plugins-config-volume\") pod \"c7b624cf-d772-4725-bad1-74d15fe3626f\" (UID: \"c7b624cf-d772-4725-bad1-74d15fe3626f\") " Oct 30 14:21:16.701485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.701487 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b624cf-d772-4725-bad1-74d15fe3626f-kube-api-access-c9jr8\" (UniqueName: \"kubernetes.io/projected/c7b624cf-d772-4725-bad1-74d15fe3626f-kube-api-access-c9jr8\") pod \"c7b624cf-d772-4725-bad1-74d15fe3626f\" (UID: \"c7b624cf-d772-4725-bad1-74d15fe3626f\") " Oct 30 14:21:16.701872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.701847 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b624cf-d772-4725-bad1-74d15fe3626f-plugins-config-volume" pod "c7b624cf-d772-4725-bad1-74d15fe3626f" (UID: "c7b624cf-d772-4725-bad1-74d15fe3626f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:16.702168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.702145 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b624cf-d772-4725-bad1-74d15fe3626f-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:16.703712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.703683 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b624cf-d772-4725-bad1-74d15fe3626f-kube-api-access-c9jr8" pod "c7b624cf-d772-4725-bad1-74d15fe3626f" (UID: "c7b624cf-d772-4725-bad1-74d15fe3626f"). InnerVolumeSpecName "kube-api-access-c9jr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:16.803732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.803702 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jr8\" (UniqueName: \"kubernetes.io/projected/c7b624cf-d772-4725-bad1-74d15fe3626f-kube-api-access-c9jr8\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:16.910917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.910835 5824 generic.go:356] "Generic (PLEG): container finished" podID="c7b624cf-d772-4725-bad1-74d15fe3626f" containerID="fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc" exitCode=0 Oct 30 14:21:16.910917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.910889 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" Oct 30 14:21:16.911128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.910911 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" event={"ID":"c7b624cf-d772-4725-bad1-74d15fe3626f","Type":"ContainerDied","Data":"fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc"} Oct 30 14:21:16.911128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.910964 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq" event={"ID":"c7b624cf-d772-4725-bad1-74d15fe3626f","Type":"ContainerDied","Data":"631a25c104fe459e169ddfe9ef7e2e012132f0821721f256b4039a10ce9f1daf"} Oct 30 14:21:16.911128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.910994 5824 scope.go:122] "RemoveContainer" containerID="fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc" Oct 30 14:21:16.913328 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.913304 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" event={"ID":"b33d70fd-8ff6-4d54-8b1d-0dae28c945d5","Type":"ContainerDied","Data":"69935234a923045ec0613587201bbea138678d53421f16d609178aaf973005f3"} Oct 30 14:21:16.913537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.913512 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c" Oct 30 14:21:16.926377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.926359 5824 scope.go:122] "RemoveContainer" containerID="fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc" Oct 30 14:21:16.926912 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:21:16.926861 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc\": container with ID starting with fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc not found: ID does not exist" containerID="fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc" Oct 30 14:21:16.927754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.927719 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc"} err="failed to get container status \"fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc\": rpc error: code = NotFound desc = could not find container \"fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc\": container with ID starting with fc481146b08be5f3a98ff4d4586ebd772fe9f2e070ec95637c7c745637b720cc not found: ID does not exist" Oct 30 14:21:16.927869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.927755 5824 scope.go:122] "RemoveContainer" containerID="ac3152cb7349407cb85b0b5fdab3a1ecb1bb6f34e1d05619f5fcad1038257361" Oct 30 14:21:16.995292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:16.995259 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c"] Oct 30 14:21:17.005746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.005711 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d74808-645df45b6c-h4f6c"] Oct 30 14:21:17.040802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.040777 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq"] Oct 30 14:21:17.071606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.071565 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b206b-55f8fbf4-lpgqq"] Oct 30 14:21:17.285563 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.285538 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:21:17.414762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.414726 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istiod-ca-cert\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.414967 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.414784 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-kube-api-access-sbb4z\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-kube-api-access-sbb4z\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.414967 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.414847 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-socket\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.415096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.414992 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-envoy\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.415178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415102 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-data\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.415267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415238 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-podinfo\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.415318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415299 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-socket" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:17.415384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415314 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-token\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-token\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.415384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415321 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istiod-ca-cert" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:17.415384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415357 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-credential-socket\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.415533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415439 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-certs\") pod \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\" (UID: \"1e55f6cb-aeb1-4628-af45-1677d1734c8b\") " Oct 30 14:21:17.415586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415528 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-data" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:17.415902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.415733 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-credential-socket" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:17.416090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.416053 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-certs" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:17.418021 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.417996 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.418221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.418028 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.418221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.418044 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.418221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.418058 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.418221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.418072 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.418855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.418821 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-token" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:17.419272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.419249 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-kube-api-access-sbb4z" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "kube-api-access-sbb4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:17.419411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.419391 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-envoy" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:17.419963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.419942 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-podinfo" pod "1e55f6cb-aeb1-4628-af45-1677d1734c8b" (UID: "1e55f6cb-aeb1-4628-af45-1677d1734c8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:21:17.520263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.519508 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.520263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.519545 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.520263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.519563 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.520263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.519580 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbb4z\" (UniqueName: \"kubernetes.io/projected/1e55f6cb-aeb1-4628-af45-1677d1734c8b-kube-api-access-sbb4z\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:17.606389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.604625 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:17.606389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.606227 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:17.635214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.628555 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:21:17.932834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.932797 5824 generic.go:356] "Generic (PLEG): container finished" podID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerID="5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a" exitCode=0 Oct 30 14:21:17.933037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.932939 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" event={"ID":"1e55f6cb-aeb1-4628-af45-1677d1734c8b","Type":"ContainerDied","Data":"5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a"} Oct 30 14:21:17.933037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.932988 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" event={"ID":"1e55f6cb-aeb1-4628-af45-1677d1734c8b","Type":"ContainerDied","Data":"6d1079adb8726f117301e4543d84b6c1812b99cfc45f757b1b0754e9461a463a"} Oct 30 14:21:17.933037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.933014 5824 scope.go:122] "RemoveContainer" containerID="5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a" Oct 30 14:21:17.933154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.933140 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s" Oct 30 14:21:17.947589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.947571 5824 scope.go:122] "RemoveContainer" containerID="5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a" Oct 30 14:21:17.947886 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:21:17.947859 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a\": container with ID starting with 5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a not found: ID does not exist" containerID="5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a" Oct 30 14:21:17.947960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.947886 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a"} err="failed to get container status \"5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a\": rpc error: code = NotFound desc = could not find container \"5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a\": container with ID starting with 5f308f1abd97b66b9557eb12320c6e21fdefd28f4721f938d9ed4ab6822ee21a not found: ID does not exist" Oct 30 14:21:17.993970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:17.993944 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s"] Oct 30 14:21:18.001181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:18.001154 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-p7d9s"] Oct 30 14:21:18.319299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:18.319230 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" path="/var/lib/kubelet/pods/1e55f6cb-aeb1-4628-af45-1677d1734c8b/volumes" Oct 30 14:21:18.319665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:18.319650 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33d70fd-8ff6-4d54-8b1d-0dae28c945d5" path="/var/lib/kubelet/pods/b33d70fd-8ff6-4d54-8b1d-0dae28c945d5/volumes" Oct 30 14:21:18.320034 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:18.320021 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b624cf-d772-4725-bad1-74d15fe3626f" path="/var/lib/kubelet/pods/c7b624cf-d772-4725-bad1-74d15fe3626f/volumes" Oct 30 14:21:24.012106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.012070 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv"] Oct 30 14:21:24.013383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013355 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b33d70fd-8ff6-4d54-8b1d-0dae28c945d5" containerName="payload-pre-processing" Oct 30 14:21:24.013383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013382 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33d70fd-8ff6-4d54-8b1d-0dae28c945d5" containerName="payload-pre-processing" Oct 30 14:21:24.013578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013451 5824 cpu_manager.go:414] "containerMap: removing container" podUID="2a1fb9bc-41d4-46e2-b760-4c370821dd4e" containerName="payload-processing" Oct 30 14:21:24.013578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013459 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1fb9bc-41d4-46e2-b760-4c370821dd4e" containerName="payload-processing" Oct 30 14:21:24.013578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013478 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" Oct 30 14:21:24.013578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013486 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" Oct 30 14:21:24.013578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013530 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c7b624cf-d772-4725-bad1-74d15fe3626f" containerName="payload-processing" Oct 30 14:21:24.013578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013537 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b624cf-d772-4725-bad1-74d15fe3626f" containerName="payload-processing" Oct 30 14:21:24.013878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013733 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b624cf-d772-4725-bad1-74d15fe3626f" containerName="payload-processing" Oct 30 14:21:24.013878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013754 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33d70fd-8ff6-4d54-8b1d-0dae28c945d5" containerName="payload-pre-processing" Oct 30 14:21:24.013878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013772 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1fb9bc-41d4-46e2-b760-4c370821dd4e" containerName="payload-processing" Oct 30 14:21:24.013878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.013783 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e55f6cb-aeb1-4628-af45-1677d1734c8b" containerName="istio-proxy" Oct 30 14:21:24.018525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.018501 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.038462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.038440 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv"] Oct 30 14:21:24.047913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.047887 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ca02c9-openshift-default-dockercfg-4hqcf\"" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175206 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-data\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175277 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-envoy\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175368 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d32f51-9414-44e3-8ea0-556d90423045-istio-podinfo\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175425 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8q2\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-kube-api-access-9s8q2\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175504 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d32f51-9414-44e3-8ea0-556d90423045-istiod-ca-cert\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175614 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-socket\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175696 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-credential-socket\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175776 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-istio-token\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.177218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.175878 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-certs\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.277834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.277753 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-certs\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.277834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.277805 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-data\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.277935 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-envoy\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278040 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d32f51-9414-44e3-8ea0-556d90423045-istio-podinfo\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278099 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8q2\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-kube-api-access-9s8q2\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278146 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-certs\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278176 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d32f51-9414-44e3-8ea0-556d90423045-istiod-ca-cert\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278183 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-data\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278344 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-socket\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278433 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-credential-socket\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278513 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-istio-token\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278669 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-socket\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278843 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-credential-socket\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.278955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.278934 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d32f51-9414-44e3-8ea0-556d90423045-istiod-ca-cert\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.280632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.280608 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-envoy\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.280816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.280800 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d32f51-9414-44e3-8ea0-556d90423045-istio-podinfo\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.306288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.306252 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8q2\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-kube-api-access-9s8q2\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.306716 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.306692 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-istio-token\") pod \"e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.338791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.338761 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:24.671842 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:21:24.671810 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d32f51_9414_44e3_8ea0_556d90423045.slice/crio-2ed20396dc77e4c5489eb7c7ec2c144cebb25cefef071fac30092f6c3c01eb4f WatchSource:0}: Error finding container 2ed20396dc77e4c5489eb7c7ec2c144cebb25cefef071fac30092f6c3c01eb4f: Status 404 returned error can't find the container with id 2ed20396dc77e4c5489eb7c7ec2c144cebb25cefef071fac30092f6c3c01eb4f Oct 30 14:21:24.673734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.673713 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv"] Oct 30 14:21:24.674933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.674900 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:24.675016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.674969 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:24.675050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:24.674999 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:25.007002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:25.006966 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" event={"ID":"03d32f51-9414-44e3-8ea0-556d90423045","Type":"ContainerStarted","Data":"f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c"} Oct 30 14:21:25.007251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:25.007021 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" event={"ID":"03d32f51-9414-44e3-8ea0-556d90423045","Type":"ContainerStarted","Data":"2ed20396dc77e4c5489eb7c7ec2c144cebb25cefef071fac30092f6c3c01eb4f"} Oct 30 14:21:25.072476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:25.072424 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" podStartSLOduration=2.072410099 podStartE2EDuration="2.072410099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:25.070063483 +0000 UTC m=+1914.870568928" watchObservedRunningTime="2026-10-30 14:21:25.072410099 +0000 UTC m=+1914.872915605" Oct 30 14:21:25.339712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:25.339635 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:25.345450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:25.345419 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:26.014987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:26.014957 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:26.015864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:26.015845 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:21:29.681942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.681905 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h"] Oct 30 14:21:29.686881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.686851 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:29.707475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.707438 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h"] Oct 30 14:21:29.715935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.715909 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ca02c9\"" Oct 30 14:21:29.716062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.715935 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ca02c9-dockercfg-p2hr9\"" Oct 30 14:21:29.778925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.778885 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtx2\" (UniqueName: \"kubernetes.io/projected/d2df3c67-41d4-457c-8115-c6449e3d5fe8-kube-api-access-5wtx2\") pod \"payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h\" (UID: \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:29.779207 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.779128 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df3c67-41d4-457c-8115-c6449e3d5fe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h\" (UID: \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:29.880414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.880370 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df3c67-41d4-457c-8115-c6449e3d5fe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h\" (UID: \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:29.880612 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.880581 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtx2\" (UniqueName: \"kubernetes.io/projected/d2df3c67-41d4-457c-8115-c6449e3d5fe8-kube-api-access-5wtx2\") pod \"payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h\" (UID: \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:29.881235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.881157 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df3c67-41d4-457c-8115-c6449e3d5fe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h\" (UID: \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:29.908144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:29.908115 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtx2\" (UniqueName: \"kubernetes.io/projected/d2df3c67-41d4-457c-8115-c6449e3d5fe8-kube-api-access-5wtx2\") pod \"payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h\" (UID: \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:30.004260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:30.002208 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:30.333113 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:21:30.333083 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2df3c67_41d4_457c_8115_c6449e3d5fe8.slice/crio-a6be0750a7b42ad8a9a352f89316a108e10d2e4f614c297c61fff4907792c5bb WatchSource:0}: Error finding container a6be0750a7b42ad8a9a352f89316a108e10d2e4f614c297c61fff4907792c5bb: Status 404 returned error can't find the container with id a6be0750a7b42ad8a9a352f89316a108e10d2e4f614c297c61fff4907792c5bb Oct 30 14:21:30.333843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:30.333818 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h"] Oct 30 14:21:31.063240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:31.063207 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" event={"ID":"d2df3c67-41d4-457c-8115-c6449e3d5fe8","Type":"ContainerStarted","Data":"9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6"} Oct 30 14:21:31.063593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:31.063256 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" event={"ID":"d2df3c67-41d4-457c-8115-c6449e3d5fe8","Type":"ContainerStarted","Data":"a6be0750a7b42ad8a9a352f89316a108e10d2e4f614c297c61fff4907792c5bb"} Oct 30 14:21:31.063593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:31.063359 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:31.119847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:31.119802 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" podStartSLOduration=2.11979083 podStartE2EDuration="2.11979083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:31.116872116 +0000 UTC m=+1920.917377557" watchObservedRunningTime="2026-10-30 14:21:31.11979083 +0000 UTC m=+1920.920296273" Oct 30 14:21:33.650854 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:21:33.650817 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:21:42.073505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:42.073473 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:21:42.394019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:42.393930 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc"] Oct 30 14:21:42.402956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:42.396659 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" containerID="cri-o://25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114" gracePeriod=30 Oct 30 14:21:47.366443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.366406 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:21:47.366825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.366470 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:21:47.718113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.718086 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:21:47.889569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889539 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-socket\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.889730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889594 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-data\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.889730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889615 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7494b496-6844-484f-ab94-65830791c8a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7494b496-6844-484f-ab94-65830791c8a7-istiod-ca-cert\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.889730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889636 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-istio-token\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-istio-token\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.889730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889724 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7494b496-6844-484f-ab94-65830791c8a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7494b496-6844-484f-ab94-65830791c8a7-istio-podinfo\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.889886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889813 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-kube-api-access-lzqqt\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-kube-api-access-lzqqt\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.889886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889824 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-data" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:47.889886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889841 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-socket" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:47.890001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.889987 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-credential-socket\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.890037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890023 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7494b496-6844-484f-ab94-65830791c8a7-istiod-ca-cert" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:21:47.890084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890058 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-certs\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.890235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890142 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-envoy\") pod \"7494b496-6844-484f-ab94-65830791c8a7\" (UID: \"7494b496-6844-484f-ab94-65830791c8a7\") " Oct 30 14:21:47.890390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890228 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-credential-socket" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:47.890390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890376 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-certs" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:47.890778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890757 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:47.890778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890778 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:47.890943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890788 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:47.890943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890796 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:47.890943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.890808 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7494b496-6844-484f-ab94-65830791c8a7-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:47.892107 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.892079 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-istio-token" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:47.892287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.892112 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7494b496-6844-484f-ab94-65830791c8a7-istio-podinfo" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:21:47.892488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.892465 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-kube-api-access-lzqqt" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "kube-api-access-lzqqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:21:47.892559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.892483 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-envoy" pod "7494b496-6844-484f-ab94-65830791c8a7" (UID: "7494b496-6844-484f-ab94-65830791c8a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:21:47.992688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.992603 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:47.992688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.992637 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7494b496-6844-484f-ab94-65830791c8a7-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:47.992688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.992648 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzqqt\" (UniqueName: \"kubernetes.io/projected/7494b496-6844-484f-ab94-65830791c8a7-kube-api-access-lzqqt\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:47.992688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:47.992671 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7494b496-6844-484f-ab94-65830791c8a7-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:21:48.267863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.267778 5824 generic.go:356] "Generic (PLEG): container finished" podID="7494b496-6844-484f-ab94-65830791c8a7" containerID="25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114" exitCode=0 Oct 30 14:21:48.267863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.267826 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" Oct 30 14:21:48.268042 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.267826 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" event={"ID":"7494b496-6844-484f-ab94-65830791c8a7","Type":"ContainerDied","Data":"25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114"} Oct 30 14:21:48.268042 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.267945 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc" event={"ID":"7494b496-6844-484f-ab94-65830791c8a7","Type":"ContainerDied","Data":"99fbb69b5fab9cf10397fa34015df976394630c748f671dc1555ae200c9ea8c3"} Oct 30 14:21:48.268042 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.267967 5824 scope.go:122] "RemoveContainer" containerID="25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114" Oct 30 14:21:48.280746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.280729 5824 scope.go:122] "RemoveContainer" containerID="30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93" Oct 30 14:21:48.290685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.290669 5824 scope.go:122] "RemoveContainer" containerID="25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114" Oct 30 14:21:48.290943 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:21:48.290925 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114\": container with ID starting with 25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114 not found: ID does not exist" containerID="25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114" Oct 30 14:21:48.290989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.290956 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114"} err="failed to get container status \"25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114\": rpc error: code = NotFound desc = could not find container \"25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114\": container with ID starting with 25f95c58ad8c7921d3b0367461cec24f210d7e5f5f5bfbe8946b9537c0579114 not found: ID does not exist" Oct 30 14:21:48.290989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.290974 5824 scope.go:122] "RemoveContainer" containerID="30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93" Oct 30 14:21:48.291250 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:21:48.291226 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93\": container with ID starting with 30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93 not found: ID does not exist" containerID="30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93" Oct 30 14:21:48.291364 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.291253 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93"} err="failed to get container status \"30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93\": rpc error: code = NotFound desc = could not find container \"30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93\": container with ID starting with 30cc63347c633c6961a176ba346041c5be0deac660ad0ad5d3a6ad758918ed93 not found: ID does not exist" Oct 30 14:21:48.329114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.329090 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc"] Oct 30 14:21:48.342067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:48.342046 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-mv7fc"] Oct 30 14:21:50.323280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:50.323249 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7494b496-6844-484f-ab94-65830791c8a7" path="/var/lib/kubelet/pods/7494b496-6844-484f-ab94-65830791c8a7/volumes" Oct 30 14:21:55.350153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.350117 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd"] Oct 30 14:21:55.350969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.350950 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" Oct 30 14:21:55.351025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.350970 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" Oct 30 14:21:55.351025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.351019 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" Oct 30 14:21:55.351025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.351024 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" Oct 30 14:21:55.351222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.351207 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" Oct 30 14:21:55.351265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.351234 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7494b496-6844-484f-ab94-65830791c8a7" containerName="istio-proxy" Oct 30 14:21:55.354062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.354046 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.380338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.380218 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd"] Oct 30 14:21:55.414162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.414127 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.414330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.414259 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.414330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.414310 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.414460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.414340 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.414522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.414455 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcgf\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-kube-api-access-bbcgf\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.415027 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.415003 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.415161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.415050 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25647b94-c002-4bf6-9a77-10e5723cdec4-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.415247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.415175 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.415307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.415247 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.516569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516537 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.516748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516578 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25647b94-c002-4bf6-9a77-10e5723cdec4-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.516748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516629 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.516748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516655 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.516748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516716 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516763 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516830 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516914 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516960 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcgf\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-kube-api-access-bbcgf\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.516974 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.517275 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-certs\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.517279 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-credential-socket\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.517401 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-data\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.517518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.517482 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25647b94-c002-4bf6-9a77-10e5723cdec4-istiod-ca-cert\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.519530 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.519488 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-envoy\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.520001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.519970 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-podinfo\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.546028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.545989 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-token\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.546394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.546365 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcgf\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-kube-api-access-bbcgf\") pod \"e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:55.669009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:55.668947 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:56.021960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:56.021916 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:56.022103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:56.021995 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:56.022103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:56.022032 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:21:56.022379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:56.022352 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd"] Oct 30 14:21:56.341295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:56.341100 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" event={"ID":"25647b94-c002-4bf6-9a77-10e5723cdec4","Type":"ContainerStarted","Data":"d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3"} Oct 30 14:21:56.341295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:56.341165 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" event={"ID":"25647b94-c002-4bf6-9a77-10e5723cdec4","Type":"ContainerStarted","Data":"6c94de79f6dfde71ba180e7f5d1d3ac0abd52cdbcb804ea887c773d2b2ec181c"} Oct 30 14:21:56.393161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:56.393112 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" podStartSLOduration=1.393098191 podStartE2EDuration="1.393098191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:21:56.390139885 +0000 UTC m=+1946.190645329" watchObservedRunningTime="2026-10-30 14:21:56.393098191 +0000 UTC m=+1946.193603634" Oct 30 14:21:56.672266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:56.669283 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:21:57.675326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:57.675286 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 14:21:57.675938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:57.675907 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 14:21:58.669761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:58.669720 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 14:21:58.669974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:58.669795 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 14:21:59.669855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:59.669824 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 14:21:59.670344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:21:59.669885 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 14:22:00.689213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:00.688414 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 14:22:00.689213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:00.688490 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 14:22:01.672198 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:01.669642 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 14:22:01.672198 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:01.669704 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 14:22:02.695939 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:02.695905 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:22:02.697884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:02.697861 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:22:02.701618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:02.701593 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:22:03.642459 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:03.642428 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:22:20.472427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:20.472391 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b"] Oct 30 14:22:20.513443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:20.513415 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4"] Oct 30 14:22:25.403795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:25.403757 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj"] Oct 30 14:22:25.438501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:25.438469 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b"] Oct 30 14:22:25.473446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:25.473406 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" podUID="3edfffcf-6f39-4b5d-b388-0a189dc55d95" containerName="payload-processing" containerID="cri-o://7234dacb48fa18d963ae89acb5e8fd3ea419eb180c472294f01d8c0a921a97dc" gracePeriod=25 Oct 30 14:22:25.514441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:25.514380 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" podUID="027d8eba-cc3e-4f14-8c89-767e1ef25ea7" containerName="payload-pre-processing" containerID="cri-o://cfe01f4d7dcaedc331f275686f26a9665764438c67619dc582f8d5a8e9d733ab" gracePeriod=25 Oct 30 14:22:25.732607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:25.732567 5824 generic.go:356] "Generic (PLEG): container finished" podID="3edfffcf-6f39-4b5d-b388-0a189dc55d95" containerID="7234dacb48fa18d963ae89acb5e8fd3ea419eb180c472294f01d8c0a921a97dc" exitCode=0 Oct 30 14:22:25.733699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:25.733613 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" event={"ID":"3edfffcf-6f39-4b5d-b388-0a189dc55d95","Type":"ContainerDied","Data":"7234dacb48fa18d963ae89acb5e8fd3ea419eb180c472294f01d8c0a921a97dc"} Oct 30 14:22:25.736657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:25.736632 5824 generic.go:356] "Generic (PLEG): container finished" podID="027d8eba-cc3e-4f14-8c89-767e1ef25ea7" containerID="cfe01f4d7dcaedc331f275686f26a9665764438c67619dc582f8d5a8e9d733ab" exitCode=0 Oct 30 14:22:25.736803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:25.736761 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" event={"ID":"027d8eba-cc3e-4f14-8c89-767e1ef25ea7","Type":"ContainerDied","Data":"cfe01f4d7dcaedc331f275686f26a9665764438c67619dc582f8d5a8e9d733ab"} Oct 30 14:22:26.015144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.015118 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:22:26.019070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.019050 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:22:26.096467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.094142 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edfffcf-6f39-4b5d-b388-0a189dc55d95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edfffcf-6f39-4b5d-b388-0a189dc55d95-plugins-config-volume\") pod \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\" (UID: \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\") " Oct 30 14:22:26.096467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.094401 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-kube-api-access-4ksxv\" (UniqueName: \"kubernetes.io/projected/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-kube-api-access-4ksxv\") pod \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\" (UID: \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\") " Oct 30 14:22:26.096467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.094439 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edfffcf-6f39-4b5d-b388-0a189dc55d95-kube-api-access-x2crk\" (UniqueName: \"kubernetes.io/projected/3edfffcf-6f39-4b5d-b388-0a189dc55d95-kube-api-access-x2crk\") pod \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\" (UID: \"3edfffcf-6f39-4b5d-b388-0a189dc55d95\") " Oct 30 14:22:26.096467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.094482 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-plugins-config-volume\") pod \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\" (UID: \"027d8eba-cc3e-4f14-8c89-767e1ef25ea7\") " Oct 30 14:22:26.096467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.095726 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-plugins-config-volume" pod "027d8eba-cc3e-4f14-8c89-767e1ef25ea7" (UID: "027d8eba-cc3e-4f14-8c89-767e1ef25ea7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:26.096467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.096129 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edfffcf-6f39-4b5d-b388-0a189dc55d95-plugins-config-volume" pod "3edfffcf-6f39-4b5d-b388-0a189dc55d95" (UID: "3edfffcf-6f39-4b5d-b388-0a189dc55d95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:26.106209 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.100732 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-kube-api-access-4ksxv" pod "027d8eba-cc3e-4f14-8c89-767e1ef25ea7" (UID: "027d8eba-cc3e-4f14-8c89-767e1ef25ea7"). InnerVolumeSpecName "kube-api-access-4ksxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:26.120222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.118400 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edfffcf-6f39-4b5d-b388-0a189dc55d95-kube-api-access-x2crk" pod "3edfffcf-6f39-4b5d-b388-0a189dc55d95" (UID: "3edfffcf-6f39-4b5d-b388-0a189dc55d95"). InnerVolumeSpecName "kube-api-access-x2crk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:26.199212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.198210 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ksxv\" (UniqueName: \"kubernetes.io/projected/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-kube-api-access-4ksxv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:22:26.199212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.198248 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2crk\" (UniqueName: \"kubernetes.io/projected/3edfffcf-6f39-4b5d-b388-0a189dc55d95-kube-api-access-x2crk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:22:26.199212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.198265 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027d8eba-cc3e-4f14-8c89-767e1ef25ea7-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:22:26.199212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.198278 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3edfffcf-6f39-4b5d-b388-0a189dc55d95-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:22:26.768941 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.768910 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" Oct 30 14:22:26.769435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.768926 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4" event={"ID":"027d8eba-cc3e-4f14-8c89-767e1ef25ea7","Type":"ContainerDied","Data":"3e504b2e6b2d3e6899e43ab9d6ebff703d1071859a2f2b06fc901025282c1b2c"} Oct 30 14:22:26.769435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.769006 5824 scope.go:122] "RemoveContainer" containerID="cfe01f4d7dcaedc331f275686f26a9665764438c67619dc582f8d5a8e9d733ab" Oct 30 14:22:26.771084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.771065 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" Oct 30 14:22:26.771361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.771096 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b" event={"ID":"3edfffcf-6f39-4b5d-b388-0a189dc55d95","Type":"ContainerDied","Data":"c6822e1e79a5ed6512bd03e7a3f37778024771356ebecbc07350b224b2e8c349"} Oct 30 14:22:26.781577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.781560 5824 scope.go:122] "RemoveContainer" containerID="7234dacb48fa18d963ae89acb5e8fd3ea419eb180c472294f01d8c0a921a97dc" Oct 30 14:22:26.828928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.828905 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4"] Oct 30 14:22:26.868211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.868021 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2a2835-5b9b7c4cf5-m54k4"] Oct 30 14:22:26.910124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.909133 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b"] Oct 30 14:22:26.913702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:26.913672 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a2835-c75b7cc9d-shj8b"] Oct 30 14:22:28.325246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:28.325211 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027d8eba-cc3e-4f14-8c89-767e1ef25ea7" path="/var/lib/kubelet/pods/027d8eba-cc3e-4f14-8c89-767e1ef25ea7/volumes" Oct 30 14:22:28.325760 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:28.325645 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edfffcf-6f39-4b5d-b388-0a189dc55d95" path="/var/lib/kubelet/pods/3edfffcf-6f39-4b5d-b388-0a189dc55d95/volumes" Oct 30 14:22:30.404474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.404428 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" podUID="bd3ed73e-5532-4eea-8e95-cd2c95c3c235" containerName="payload-processing" containerID="cri-o://cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907" gracePeriod=25 Oct 30 14:22:30.439180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.439130 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" podUID="950244f2-2998-4c57-8ec7-6a077ad9df7e" containerName="payload-pre-processing" containerID="cri-o://b0770d278c8e2cbba0ecabe1a8b2dd243ff3378da12cc72c4b2e39d58624644f" gracePeriod=25 Oct 30 14:22:30.539597 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:30.539558 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950244f2_2998_4c57_8ec7_6a077ad9df7e.slice/crio-b0770d278c8e2cbba0ecabe1a8b2dd243ff3378da12cc72c4b2e39d58624644f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:22:30.818696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.818658 5824 generic.go:356] "Generic (PLEG): container finished" podID="bd3ed73e-5532-4eea-8e95-cd2c95c3c235" containerID="cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907" exitCode=0 Oct 30 14:22:30.818959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.818729 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" event={"ID":"bd3ed73e-5532-4eea-8e95-cd2c95c3c235","Type":"ContainerDied","Data":"cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907"} Oct 30 14:22:30.821145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.821117 5824 generic.go:356] "Generic (PLEG): container finished" podID="950244f2-2998-4c57-8ec7-6a077ad9df7e" containerID="b0770d278c8e2cbba0ecabe1a8b2dd243ff3378da12cc72c4b2e39d58624644f" exitCode=0 Oct 30 14:22:30.821344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.821289 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" event={"ID":"950244f2-2998-4c57-8ec7-6a077ad9df7e","Type":"ContainerDied","Data":"b0770d278c8e2cbba0ecabe1a8b2dd243ff3378da12cc72c4b2e39d58624644f"} Oct 30 14:22:30.911699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.911671 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:22:30.967437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.967410 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-plugins-config-volume\") pod \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\" (UID: \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\") " Oct 30 14:22:30.967642 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.967624 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-kube-api-access-knmp7\" (UniqueName: \"kubernetes.io/projected/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-kube-api-access-knmp7\") pod \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\" (UID: \"bd3ed73e-5532-4eea-8e95-cd2c95c3c235\") " Oct 30 14:22:30.967836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.967813 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-plugins-config-volume" pod "bd3ed73e-5532-4eea-8e95-cd2c95c3c235" (UID: "bd3ed73e-5532-4eea-8e95-cd2c95c3c235"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:30.968220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.968172 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:22:30.970106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:30.970075 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-kube-api-access-knmp7" pod "bd3ed73e-5532-4eea-8e95-cd2c95c3c235" (UID: "bd3ed73e-5532-4eea-8e95-cd2c95c3c235"). InnerVolumeSpecName "kube-api-access-knmp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:31.069668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.069643 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knmp7\" (UniqueName: \"kubernetes.io/projected/bd3ed73e-5532-4eea-8e95-cd2c95c3c235-kube-api-access-knmp7\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:22:31.320023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.319948 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:22:31.373743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.373703 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950244f2-2998-4c57-8ec7-6a077ad9df7e-kube-api-access-sswps\" (UniqueName: \"kubernetes.io/projected/950244f2-2998-4c57-8ec7-6a077ad9df7e-kube-api-access-sswps\") pod \"950244f2-2998-4c57-8ec7-6a077ad9df7e\" (UID: \"950244f2-2998-4c57-8ec7-6a077ad9df7e\") " Oct 30 14:22:31.373943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.373924 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950244f2-2998-4c57-8ec7-6a077ad9df7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950244f2-2998-4c57-8ec7-6a077ad9df7e-plugins-config-volume\") pod \"950244f2-2998-4c57-8ec7-6a077ad9df7e\" (UID: \"950244f2-2998-4c57-8ec7-6a077ad9df7e\") " Oct 30 14:22:31.374422 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.374393 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950244f2-2998-4c57-8ec7-6a077ad9df7e-plugins-config-volume" pod "950244f2-2998-4c57-8ec7-6a077ad9df7e" (UID: "950244f2-2998-4c57-8ec7-6a077ad9df7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:22:31.375430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.375407 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950244f2-2998-4c57-8ec7-6a077ad9df7e-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:22:31.376223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.376201 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950244f2-2998-4c57-8ec7-6a077ad9df7e-kube-api-access-sswps" pod "950244f2-2998-4c57-8ec7-6a077ad9df7e" (UID: "950244f2-2998-4c57-8ec7-6a077ad9df7e"). InnerVolumeSpecName "kube-api-access-sswps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:22:31.476627 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.476598 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sswps\" (UniqueName: \"kubernetes.io/projected/950244f2-2998-4c57-8ec7-6a077ad9df7e-kube-api-access-sswps\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:22:31.836869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.836830 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" event={"ID":"bd3ed73e-5532-4eea-8e95-cd2c95c3c235","Type":"ContainerDied","Data":"67f4d83b6c7314094b6e6cc755733f54bad0540b03dd25f707f2e3b84adb65a6"} Oct 30 14:22:31.837136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.836842 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj" Oct 30 14:22:31.837136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.836937 5824 scope.go:122] "RemoveContainer" containerID="cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907" Oct 30 14:22:31.838495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.838475 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" Oct 30 14:22:31.838594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.838474 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b" event={"ID":"950244f2-2998-4c57-8ec7-6a077ad9df7e","Type":"ContainerDied","Data":"8533a77d4b56f73974cc7ea528247655fff39e86d10d697e4b36caa0af382e83"} Oct 30 14:22:31.849162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.849136 5824 scope.go:122] "RemoveContainer" containerID="b0770d278c8e2cbba0ecabe1a8b2dd243ff3378da12cc72c4b2e39d58624644f" Oct 30 14:22:31.898722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.898689 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj"] Oct 30 14:22:31.909913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.909894 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c4b82-55c456d68d-4gnsj"] Oct 30 14:22:31.950567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.950544 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b"] Oct 30 14:22:31.954517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:31.954497 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c4b82-547d7565c9-pkl7b"] Oct 30 14:22:32.327825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:32.319589 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950244f2-2998-4c57-8ec7-6a077ad9df7e" path="/var/lib/kubelet/pods/950244f2-2998-4c57-8ec7-6a077ad9df7e/volumes" Oct 30 14:22:32.327825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:32.320002 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3ed73e-5532-4eea-8e95-cd2c95c3c235" path="/var/lib/kubelet/pods/bd3ed73e-5532-4eea-8e95-cd2c95c3c235/volumes" Oct 30 14:22:32.849044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:32.849015 5824 generic.go:356] "Generic (PLEG): container finished" podID="03d32f51-9414-44e3-8ea0-556d90423045" containerID="f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c" exitCode=0 Oct 30 14:22:32.849389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:32.849094 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" event={"ID":"03d32f51-9414-44e3-8ea0-556d90423045","Type":"ContainerDied","Data":"f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c"} Oct 30 14:22:32.849597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:32.849581 5824 scope.go:122] "RemoveContainer" containerID="f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c" Oct 30 14:22:32.851435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:32.851403 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:32.851531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:32.851475 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:32.851531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:32.851508 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:22:33.645828 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:33.645794 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:22:33.867271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:33.867227 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" event={"ID":"03d32f51-9414-44e3-8ea0-556d90423045","Type":"ContainerStarted","Data":"02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32"} Oct 30 14:22:34.340255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:34.339996 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:22:34.340255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:34.340136 5824 patch_prober.go:29] interesting pod/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 14:22:34.340255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:34.340181 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 14:22:35.368214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:35.352602 5824 patch_prober.go:29] interesting pod/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 14:22:35.368214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:35.352671 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 14:22:36.224322 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:36.224285 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:22:36.339268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:36.339151 5824 patch_prober.go:29] interesting pod/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 14:22:36.339268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:36.339253 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 14:22:37.339565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:37.339529 5824 patch_prober.go:29] interesting pod/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 14:22:37.339943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:37.339580 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 14:22:38.339903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:38.339871 5824 patch_prober.go:29] interesting pod/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 14:22:38.340313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:38.339924 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 14:22:39.343746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:39.343713 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:22:39.344231 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:39.344001 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:22:39.344791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:22:39.344775 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:22:40.559046 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:40.559011 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:22:48.576703 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:48.576616 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:22:48.576703 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:48.576638 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:22:50.575516 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:50.575475 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:22:51.102925 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:22:51.102884 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:23:00.674796 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:00.674761 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:23:03.662459 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:03.662420 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:23:06.180436 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:06.180401 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:23:06.320625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.320594 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt"] Oct 30 14:23:06.321317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321301 5824 cpu_manager.go:414] "containerMap: removing container" podUID="950244f2-2998-4c57-8ec7-6a077ad9df7e" containerName="payload-pre-processing" Oct 30 14:23:06.321366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321318 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950244f2-2998-4c57-8ec7-6a077ad9df7e" containerName="payload-pre-processing" Oct 30 14:23:06.321366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321346 5824 cpu_manager.go:414] "containerMap: removing container" podUID="bd3ed73e-5532-4eea-8e95-cd2c95c3c235" containerName="payload-processing" Oct 30 14:23:06.321366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321353 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3ed73e-5532-4eea-8e95-cd2c95c3c235" containerName="payload-processing" Oct 30 14:23:06.321464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321369 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3edfffcf-6f39-4b5d-b388-0a189dc55d95" containerName="payload-processing" Oct 30 14:23:06.321464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321374 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edfffcf-6f39-4b5d-b388-0a189dc55d95" containerName="payload-processing" Oct 30 14:23:06.321464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321390 5824 cpu_manager.go:414] "containerMap: removing container" podUID="027d8eba-cc3e-4f14-8c89-767e1ef25ea7" containerName="payload-pre-processing" Oct 30 14:23:06.321464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321395 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027d8eba-cc3e-4f14-8c89-767e1ef25ea7" containerName="payload-pre-processing" Oct 30 14:23:06.321584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321514 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="950244f2-2998-4c57-8ec7-6a077ad9df7e" containerName="payload-pre-processing" Oct 30 14:23:06.321584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321526 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3ed73e-5532-4eea-8e95-cd2c95c3c235" containerName="payload-processing" Oct 30 14:23:06.321584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321534 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edfffcf-6f39-4b5d-b388-0a189dc55d95" containerName="payload-processing" Oct 30 14:23:06.321584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.321542 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="027d8eba-cc3e-4f14-8c89-767e1ef25ea7" containerName="payload-pre-processing" Oct 30 14:23:06.324720 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.324695 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.371099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.366522 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ea9ed2-openshift-default-dockercfg-fk8hk\"" Oct 30 14:23:06.394421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.394391 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt"] Oct 30 14:23:06.499940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.499866 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.499940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.499922 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac379e-c863-4eac-af03-5a727f03f70a-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.500154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.499969 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.500263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.500237 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.500325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.500312 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac379e-c863-4eac-af03-5a727f03f70a-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.500512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.500470 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9bn\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-kube-api-access-4h9bn\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.500569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.500552 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.500626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.500588 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.500758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.500736 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602278 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602336 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac379e-c863-4eac-af03-5a727f03f70a-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602402 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9bn\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-kube-api-access-4h9bn\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602445 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602473 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602507 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602636 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602680 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac379e-c863-4eac-af03-5a727f03f70a-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602699 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.602845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.602714 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.603053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.603022 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.603239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.603161 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.603239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.603215 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.603474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.603452 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac379e-c863-4eac-af03-5a727f03f70a-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.605031 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.605011 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.605607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.605589 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac379e-c863-4eac-af03-5a727f03f70a-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.629907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.629886 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9bn\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-kube-api-access-4h9bn\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.630025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.630002 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.640010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.639989 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:06.961668 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:23:06.961642 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefac379e_c863_4eac_af03_5a727f03f70a.slice/crio-a71cb3b6f3b66e59146f9e487e8cef259ffe000e1d56c513ac46cda3f468e22d WatchSource:0}: Error finding container a71cb3b6f3b66e59146f9e487e8cef259ffe000e1d56c513ac46cda3f468e22d: Status 404 returned error can't find the container with id a71cb3b6f3b66e59146f9e487e8cef259ffe000e1d56c513ac46cda3f468e22d Oct 30 14:23:06.962853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.962830 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt"] Oct 30 14:23:06.964521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.964486 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:06.964620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.964564 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:06.964620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:06.964613 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:07.231266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:07.230423 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" event={"ID":"efac379e-c863-4eac-af03-5a727f03f70a","Type":"ContainerStarted","Data":"104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c"} Oct 30 14:23:07.231266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:07.230509 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" event={"ID":"efac379e-c863-4eac-af03-5a727f03f70a","Type":"ContainerStarted","Data":"a71cb3b6f3b66e59146f9e487e8cef259ffe000e1d56c513ac46cda3f468e22d"} Oct 30 14:23:07.287184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:07.287115 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" podStartSLOduration=1.287090643 podStartE2EDuration="1.287090643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:07.284650221 +0000 UTC m=+2017.085155665" watchObservedRunningTime="2026-10-30 14:23:07.287090643 +0000 UTC m=+2017.087596085" Oct 30 14:23:07.640490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:07.640410 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:08.040439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:08.040410 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6"] Oct 30 14:23:08.078458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:08.078425 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ffca3b-8nlp6"] Oct 30 14:23:08.324700 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:08.324600 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c" path="/var/lib/kubelet/pods/6eb0d62e-e0a6-4be1-a1d8-3cf02cce615c/volumes" Oct 30 14:23:08.653849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:08.653762 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 14:23:08.681478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:08.681306 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 14:23:09.640611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:09.640580 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 14:23:09.640992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:09.640628 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 14:23:10.640880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:10.640851 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 14:23:10.641444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:10.640905 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 14:23:10.692568 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:10.692539 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:23:11.200644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.200607 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr"] Oct 30 14:23:11.204753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.204731 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:11.222439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.222418 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr"] Oct 30 14:23:11.232384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.232362 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d15602\"" Oct 30 14:23:11.270068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.270040 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d15602-dockercfg-7tjsj\"" Oct 30 14:23:11.303736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.303685 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e465ab52-9753-43f7-8b53-50e20ce58401-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr\" (UID: \"e465ab52-9753-43f7-8b53-50e20ce58401\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:11.303938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.303870 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2pq\" (UniqueName: \"kubernetes.io/projected/e465ab52-9753-43f7-8b53-50e20ce58401-kube-api-access-6j2pq\") pod \"payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr\" (UID: \"e465ab52-9753-43f7-8b53-50e20ce58401\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:11.416233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.415479 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e465ab52-9753-43f7-8b53-50e20ce58401-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr\" (UID: \"e465ab52-9753-43f7-8b53-50e20ce58401\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:11.416233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.415655 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2pq\" (UniqueName: \"kubernetes.io/projected/e465ab52-9753-43f7-8b53-50e20ce58401-kube-api-access-6j2pq\") pod \"payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr\" (UID: \"e465ab52-9753-43f7-8b53-50e20ce58401\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:11.422876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.417709 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e465ab52-9753-43f7-8b53-50e20ce58401-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr\" (UID: \"e465ab52-9753-43f7-8b53-50e20ce58401\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:11.569260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.568298 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2pq\" (UniqueName: \"kubernetes.io/projected/e465ab52-9753-43f7-8b53-50e20ce58401-kube-api-access-6j2pq\") pod \"payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr\" (UID: \"e465ab52-9753-43f7-8b53-50e20ce58401\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:11.641253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.641213 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 14:23:11.641734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.641280 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 14:23:11.819126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:11.819088 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:12.183909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:12.182900 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr"] Oct 30 14:23:12.185717 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:23:12.185683 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode465ab52_9753_43f7_8b53_50e20ce58401.slice/crio-28070cd35aa8b87759ad40217aae913ed71a7e4a099cfbf0d46b317fd2e0b50f WatchSource:0}: Error finding container 28070cd35aa8b87759ad40217aae913ed71a7e4a099cfbf0d46b317fd2e0b50f: Status 404 returned error can't find the container with id 28070cd35aa8b87759ad40217aae913ed71a7e4a099cfbf0d46b317fd2e0b50f Oct 30 14:23:12.576793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:12.576714 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" event={"ID":"e465ab52-9753-43f7-8b53-50e20ce58401","Type":"ContainerStarted","Data":"dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357"} Oct 30 14:23:12.576793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:12.576768 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" event={"ID":"e465ab52-9753-43f7-8b53-50e20ce58401","Type":"ContainerStarted","Data":"28070cd35aa8b87759ad40217aae913ed71a7e4a099cfbf0d46b317fd2e0b50f"} Oct 30 14:23:12.576992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:12.576938 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:12.625639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:12.624718 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" podStartSLOduration=1.624706593 podStartE2EDuration="1.624706593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:12.621914594 +0000 UTC m=+2022.422420037" watchObservedRunningTime="2026-10-30 14:23:12.624706593 +0000 UTC m=+2022.425212035" Oct 30 14:23:12.640725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:12.640694 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 14:23:12.640887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:12.640758 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 14:23:13.645001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:13.644973 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:13.645460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:13.645443 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:13.646044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:13.646026 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:23:20.817056 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:20.817007 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:23:21.086487 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:21.086404 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ed73e_5532_4eea_8e95_cd2c95c3c235.slice/crio-conmon-cacf7a56fdc91db67c7b9c186d43318ce960d5e25f0a15d8a0939dec0b460907.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:23:23.587784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:23.587755 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:23:31.378524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:31.378494 5824 scope.go:122] "RemoveContainer" containerID="b026b2bab02b7138c72c09c0a4e7ec7ccec101015c7a1f7c54c5f57c69f17d5d" Oct 30 14:23:33.645869 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:33.645836 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:23:44.467586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:44.467549 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z"] Oct 30 14:23:44.468145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:44.467759 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" containerID="cri-o://13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193" gracePeriod=30 Oct 30 14:23:44.480913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:44.480883 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd"] Oct 30 14:23:44.481094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:44.481074 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" containerID="cri-o://d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3" gracePeriod=30 Oct 30 14:23:47.608748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:47.608702 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:23:47.609148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:47.608774 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:23:47.701108 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:47.701066 5824 patch_prober.go:29] interesting pod/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:23:47.701311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:47.701167 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:23:49.811756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.811729 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:23:49.824756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.824732 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:23:49.839336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839299 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-credential-socket\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.839461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839355 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-certs\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.839461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839411 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6860cde2-f7a8-42e0-b3c9-c02000870156-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6860cde2-f7a8-42e0-b3c9-c02000870156-istiod-ca-cert\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.839461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839450 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-kube-api-access-bbcgf\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-kube-api-access-bbcgf\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.839608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839495 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-data\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.839608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839561 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-socket\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.839720 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839666 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-data\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.839720 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839714 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-podinfo\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839759 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-token\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-token\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839799 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-token\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-token\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839903 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-socket\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839933 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-envoy\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839934 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-certs" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.839967 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25647b94-c002-4bf6-9a77-10e5723cdec4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25647b94-c002-4bf6-9a77-10e5723cdec4-istiod-ca-cert\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840096 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6860cde2-f7a8-42e0-b3c9-c02000870156-istiod-ca-cert" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840173 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-data" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840408 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-podinfo\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.840731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840706 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-credential-socket\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840778 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-kube-api-access-zgbcr\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-kube-api-access-zgbcr\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840781 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-socket" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840932 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-envoy\") pod \"25647b94-c002-4bf6-9a77-10e5723cdec4\" (UID: \"25647b94-c002-4bf6-9a77-10e5723cdec4\") " Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840973 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-certs\") pod \"6860cde2-f7a8-42e0-b3c9-c02000870156\" (UID: \"6860cde2-f7a8-42e0-b3c9-c02000870156\") " Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.840992 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-data" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.841331 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-credential-socket" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.841394 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25647b94-c002-4bf6-9a77-10e5723cdec4-istiod-ca-cert" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.841537 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-credential-socket" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.841905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.841548 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-certs" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.847278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.847255 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-socket" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.848455 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.848431 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-podinfo" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:23:49.849079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.849056 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.849270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.849253 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.849379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.849366 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.849480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.849467 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6860cde2-f7a8-42e0-b3c9-c02000870156-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.849573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.849563 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.849660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.849648 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.849756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.849736 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.850088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.850066 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.850088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.850087 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25647b94-c002-4bf6-9a77-10e5723cdec4-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.850278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.850100 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.850278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.850113 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.850278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.849880 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-podinfo" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:23:49.854688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.854666 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-kube-api-access-zgbcr" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "kube-api-access-zgbcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:49.854862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.854844 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-envoy" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.855124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.855097 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-token" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:49.855280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.855262 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-envoy" pod "6860cde2-f7a8-42e0-b3c9-c02000870156" (UID: "6860cde2-f7a8-42e0-b3c9-c02000870156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:23:49.856611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.856590 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-token" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:49.875330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.875302 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-kube-api-access-bbcgf" pod "25647b94-c002-4bf6-9a77-10e5723cdec4" (UID: "25647b94-c002-4bf6-9a77-10e5723cdec4"). InnerVolumeSpecName "kube-api-access-bbcgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:23:49.953696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.953656 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.953696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.953697 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcgf\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-kube-api-access-bbcgf\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.953928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.953715 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.953928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.953730 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25647b94-c002-4bf6-9a77-10e5723cdec4-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.953928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.953744 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.953928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.953757 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6860cde2-f7a8-42e0-b3c9-c02000870156-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.953928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.953772 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbcr\" (UniqueName: \"kubernetes.io/projected/6860cde2-f7a8-42e0-b3c9-c02000870156-kube-api-access-zgbcr\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:23:49.993050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.993008 5824 generic.go:356] "Generic (PLEG): container finished" podID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerID="13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193" exitCode=0 Oct 30 14:23:49.993276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.993175 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" event={"ID":"6860cde2-f7a8-42e0-b3c9-c02000870156","Type":"ContainerDied","Data":"13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193"} Oct 30 14:23:49.993348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.993294 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" event={"ID":"6860cde2-f7a8-42e0-b3c9-c02000870156","Type":"ContainerDied","Data":"548c7fd93948de35dfe87de0b8a55d71cde3306ea48e0d5daf5e2918e22f8039"} Oct 30 14:23:49.993348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.993339 5824 scope.go:122] "RemoveContainer" containerID="13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193" Oct 30 14:23:49.993573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.993557 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z" Oct 30 14:23:49.996070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.996041 5824 generic.go:356] "Generic (PLEG): container finished" podID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerID="d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3" exitCode=0 Oct 30 14:23:49.996258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.996152 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" event={"ID":"25647b94-c002-4bf6-9a77-10e5723cdec4","Type":"ContainerDied","Data":"d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3"} Oct 30 14:23:49.996258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.996235 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" event={"ID":"25647b94-c002-4bf6-9a77-10e5723cdec4","Type":"ContainerDied","Data":"6c94de79f6dfde71ba180e7f5d1d3ac0abd52cdbcb804ea887c773d2b2ec181c"} Oct 30 14:23:49.996399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:49.996311 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd" Oct 30 14:23:50.026938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.026914 5824 scope.go:122] "RemoveContainer" containerID="13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193" Oct 30 14:23:50.027323 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:50.027303 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193\": container with ID starting with 13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193 not found: ID does not exist" containerID="13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193" Oct 30 14:23:50.027423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.027333 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193"} err="failed to get container status \"13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193\": rpc error: code = NotFound desc = could not find container \"13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193\": container with ID starting with 13a38001199b6d6326f7e43affc15ad23e8fdad6083dd9bda138c06753c7d193 not found: ID does not exist" Oct 30 14:23:50.027423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.027356 5824 scope.go:122] "RemoveContainer" containerID="d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3" Oct 30 14:23:50.037463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.037442 5824 scope.go:122] "RemoveContainer" containerID="d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3" Oct 30 14:23:50.037723 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:23:50.037702 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3\": container with ID starting with d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3 not found: ID does not exist" containerID="d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3" Oct 30 14:23:50.037798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.037729 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3"} err="failed to get container status \"d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3\": rpc error: code = NotFound desc = could not find container \"d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3\": container with ID starting with d35d57c76d75263ba826212f1fa3ffeafce3de1906861d980c85bc19db4f93b3 not found: ID does not exist" Oct 30 14:23:50.056312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.056285 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z"] Oct 30 14:23:50.059831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.059805 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-94l6z"] Oct 30 14:23:50.119479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.119449 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd"] Oct 30 14:23:50.125723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.125698 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2a2835-openshift-default-74dc4b7c9c-6lsmd"] Oct 30 14:23:50.322417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.322332 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" path="/var/lib/kubelet/pods/25647b94-c002-4bf6-9a77-10e5723cdec4/volumes" Oct 30 14:23:50.322976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:50.322956 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" path="/var/lib/kubelet/pods/6860cde2-f7a8-42e0-b3c9-c02000870156/volumes" Oct 30 14:23:51.507599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.507563 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8"] Oct 30 14:23:51.508392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.508374 5824 cpu_manager.go:414] "containerMap: removing container" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" Oct 30 14:23:51.508392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.508391 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" Oct 30 14:23:51.508531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.508409 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" Oct 30 14:23:51.508531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.508415 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" Oct 30 14:23:51.508531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.508513 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="25647b94-c002-4bf6-9a77-10e5723cdec4" containerName="istio-proxy" Oct 30 14:23:51.508531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.508522 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6860cde2-f7a8-42e0-b3c9-c02000870156" containerName="istio-proxy" Oct 30 14:23:51.512551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.512530 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.540208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.540138 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8"] Oct 30 14:23:51.552561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.552534 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds"] Oct 30 14:23:51.559392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.559370 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598255 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598323 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ae12b0-3781-48e3-a967-13e946c58fc1-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598356 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djf5l\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-kube-api-access-djf5l\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598446 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598520 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598557 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598606 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598659 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.600478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.598696 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.650235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.626255 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds"] Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718219 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd0b83-39b9-4b30-89ba-337004b917ea-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718316 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718354 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ae12b0-3781-48e3-a967-13e946c58fc1-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718390 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djf5l\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-kube-api-access-djf5l\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718460 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718509 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718582 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718612 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718652 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718706 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718736 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718793 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718835 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.718892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718861 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgzk\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-kube-api-access-7tgzk\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.719725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718938 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.719725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.718998 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.719725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.719037 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.719725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.719101 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.719725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.719672 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.745229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.720331 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ae12b0-3781-48e3-a967-13e946c58fc1-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.745229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.720919 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.753660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.753628 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.753881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.753856 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.754234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.754102 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.817896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.817816 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djf5l\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-kube-api-access-djf5l\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.821240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.821206 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgzk\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-kube-api-access-7tgzk\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.821378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.821315 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.821866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.821844 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.822441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.822217 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-credential-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.822441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.822334 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.823458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.823430 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-data\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.823907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.823875 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.824027 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.824010 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd0b83-39b9-4b30-89ba-337004b917ea-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.824154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.824139 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.824300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.824285 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.824393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.824355 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.824718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.824697 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-socket\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.824921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.824903 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-certs\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.825340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.825312 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.825712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.825686 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd0b83-39b9-4b30-89ba-337004b917ea-istiod-ca-cert\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.827009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.826990 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.827184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.827162 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-podinfo\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:51.827184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.827176 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-envoy\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.847367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.847343 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgzk\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-kube-api-access-7tgzk\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.851871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.851847 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-token\") pod \"e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:51.883532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:51.883508 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:52.127165 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.127064 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:52.215457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.215428 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds"] Oct 30 14:23:52.216573 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:23:52.216548 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdd0b83_39b9_4b30_89ba_337004b917ea.slice/crio-654eef8cbaf8c95f3a25cae34b28da0f00f9e5c8003ff855697c243e6fed1166 WatchSource:0}: Error finding container 654eef8cbaf8c95f3a25cae34b28da0f00f9e5c8003ff855697c243e6fed1166: Status 404 returned error can't find the container with id 654eef8cbaf8c95f3a25cae34b28da0f00f9e5c8003ff855697c243e6fed1166 Oct 30 14:23:52.220677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.220641 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:52.220788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.220709 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:52.220788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.220738 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:52.444102 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:23:52.444074 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ae12b0_3781_48e3_a967_13e946c58fc1.slice/crio-2b660990df8e4fdfaeb1db5cb4e1f6080f60242fdb837b192e64691a90bc56f1 WatchSource:0}: Error finding container 2b660990df8e4fdfaeb1db5cb4e1f6080f60242fdb837b192e64691a90bc56f1: Status 404 returned error can't find the container with id 2b660990df8e4fdfaeb1db5cb4e1f6080f60242fdb837b192e64691a90bc56f1 Oct 30 14:23:52.445527 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.445506 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8"] Oct 30 14:23:52.446987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.446950 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:52.447061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.447019 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:52.447061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:52.447048 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:23:53.061386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.061350 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" event={"ID":"7cdd0b83-39b9-4b30-89ba-337004b917ea","Type":"ContainerStarted","Data":"88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5"} Oct 30 14:23:53.061876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.061411 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" event={"ID":"7cdd0b83-39b9-4b30-89ba-337004b917ea","Type":"ContainerStarted","Data":"654eef8cbaf8c95f3a25cae34b28da0f00f9e5c8003ff855697c243e6fed1166"} Oct 30 14:23:53.063232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.063171 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" event={"ID":"16ae12b0-3781-48e3-a967-13e946c58fc1","Type":"ContainerStarted","Data":"e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86"} Oct 30 14:23:53.063359 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.063259 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" event={"ID":"16ae12b0-3781-48e3-a967-13e946c58fc1","Type":"ContainerStarted","Data":"2b660990df8e4fdfaeb1db5cb4e1f6080f60242fdb837b192e64691a90bc56f1"} Oct 30 14:23:53.124874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.124145 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podStartSLOduration=2.124122888 podStartE2EDuration="2.124122888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:53.11236922 +0000 UTC m=+2062.912874666" watchObservedRunningTime="2026-10-30 14:23:53.124122888 +0000 UTC m=+2062.924628332" Oct 30 14:23:53.128809 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.128765 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:23:53.134863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.134833 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 14:23:53.135106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.135071 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 14:23:53.163269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.163212 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podStartSLOduration=2.163179274 podStartE2EDuration="2.163179274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:53.162650081 +0000 UTC m=+2062.963155528" watchObservedRunningTime="2026-10-30 14:23:53.163179274 +0000 UTC m=+2062.963684720" Oct 30 14:23:53.888213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.887674 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:53.892225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.889977 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 14:23:53.892225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:53.890037 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 14:23:54.127742 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.127706 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 14:23:54.128663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.128623 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 14:23:54.303913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.303673 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh"] Oct 30 14:23:54.309000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.308967 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:54.328516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.328490 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh"] Oct 30 14:23:54.340480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.340451 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-409364-dockercfg-jjtxs\"" Oct 30 14:23:54.401215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.389671 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-409364\"" Oct 30 14:23:54.423215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.420687 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsfb\" (UniqueName: \"kubernetes.io/projected/e55fba24-dca9-4b03-b9a0-645ee11072a9-kube-api-access-ffsfb\") pod \"payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh\" (UID: \"e55fba24-dca9-4b03-b9a0-645ee11072a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:54.423215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.420778 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55fba24-dca9-4b03-b9a0-645ee11072a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh\" (UID: \"e55fba24-dca9-4b03-b9a0-645ee11072a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:54.536228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.535077 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsfb\" (UniqueName: \"kubernetes.io/projected/e55fba24-dca9-4b03-b9a0-645ee11072a9-kube-api-access-ffsfb\") pod \"payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh\" (UID: \"e55fba24-dca9-4b03-b9a0-645ee11072a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:54.536228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.535182 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55fba24-dca9-4b03-b9a0-645ee11072a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh\" (UID: \"e55fba24-dca9-4b03-b9a0-645ee11072a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:54.536228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.536170 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55fba24-dca9-4b03-b9a0-645ee11072a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh\" (UID: \"e55fba24-dca9-4b03-b9a0-645ee11072a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:54.638262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.637361 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsfb\" (UniqueName: \"kubernetes.io/projected/e55fba24-dca9-4b03-b9a0-645ee11072a9-kube-api-access-ffsfb\") pod \"payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh\" (UID: \"e55fba24-dca9-4b03-b9a0-645ee11072a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:54.890269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.884534 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 14:23:54.890269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.884592 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 14:23:54.932517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:54.932488 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:55.128140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:55.128099 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 14:23:55.128551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:55.128164 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 14:23:55.396860 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:23:55.396794 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55fba24_dca9_4b03_b9a0_645ee11072a9.slice/crio-86ec666df7279a5ed2d4fbaabc877724dbdb0c8cacc43df18122296447deb26f WatchSource:0}: Error finding container 86ec666df7279a5ed2d4fbaabc877724dbdb0c8cacc43df18122296447deb26f: Status 404 returned error can't find the container with id 86ec666df7279a5ed2d4fbaabc877724dbdb0c8cacc43df18122296447deb26f Oct 30 14:23:55.398619 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:55.398594 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh"] Oct 30 14:23:55.893241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:55.892439 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 14:23:55.893241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:55.892502 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 14:23:56.128399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:56.128360 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 14:23:56.128786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:56.128412 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 14:23:56.137920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:56.137871 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" event={"ID":"e55fba24-dca9-4b03-b9a0-645ee11072a9","Type":"ContainerStarted","Data":"93dfbe084b7a33ea9cd8df7afc4a0ab571b5731feeb5e3cbf33213e9f00da6b3"} Oct 30 14:23:56.138015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:56.137942 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" event={"ID":"e55fba24-dca9-4b03-b9a0-645ee11072a9","Type":"ContainerStarted","Data":"86ec666df7279a5ed2d4fbaabc877724dbdb0c8cacc43df18122296447deb26f"} Oct 30 14:23:56.138297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:56.138274 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:23:56.196275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:56.195957 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" podStartSLOduration=2.195940624 podStartE2EDuration="2.195940624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:23:56.190487669 +0000 UTC m=+2065.990993114" watchObservedRunningTime="2026-10-30 14:23:56.195940624 +0000 UTC m=+2065.996446067" Oct 30 14:23:56.886212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:56.885548 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 14:23:56.886212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:56.885612 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 14:23:57.128212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:57.128157 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 14:23:57.128387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:57.128234 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 14:23:57.915578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:57.915538 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 14:23:57.916398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:57.916362 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 14:23:58.128324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:58.128292 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 14:23:58.128572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:58.128551 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 14:23:58.887943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:58.887509 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 14:23:58.887943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:58.887570 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 14:23:59.128094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:59.128058 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 14:23:59.128521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:59.128113 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 14:23:59.889363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:59.889328 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:59.889746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:59.889716 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:23:59.890657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:23:59.890634 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:24:00.132098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:00.132069 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:24:00.132600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:00.132582 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:24:00.133336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:00.133313 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:24:03.651222 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:24:03.651160 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:24:07.175203 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:07.175159 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:24:31.416860 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:31.416783 5824 scope.go:122] "RemoveContainer" containerID="ce999ea31be768fe82505b4a007569e047d7e696fcda25335580e51261f759ff" Oct 30 14:24:33.643616 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:24:33.643580 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:24:42.727012 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:42.726979 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl"] Oct 30 14:24:42.727539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:42.727199 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" containerID="cri-o://b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e" gracePeriod=30 Oct 30 14:24:47.599288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:47.599248 5824 patch_prober.go:29] interesting pod/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:24:47.599677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:47.599337 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:24:48.008588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.008557 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:24:48.090142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090105 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-istio-token\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-istio-token\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090202 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-data\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090237 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be00760-ece6-49c2-97f9-26a0daab728e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be00760-ece6-49c2-97f9-26a0daab728e-istiod-ca-cert\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090275 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6be00760-ece6-49c2-97f9-26a0daab728e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be00760-ece6-49c2-97f9-26a0daab728e-istio-podinfo\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090321 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-socket\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090380 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-credential-socket\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090408 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-kube-api-access-jf4w9\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-kube-api-access-jf4w9\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090444 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-envoy\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090468 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-certs\") pod \"6be00760-ece6-49c2-97f9-26a0daab728e\" (UID: \"6be00760-ece6-49c2-97f9-26a0daab728e\") " Oct 30 14:24:48.090604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090481 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-data" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:48.090875 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090644 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-socket" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:48.090875 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090695 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-credential-socket" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:48.090984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.090925 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-certs" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:48.091169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.091143 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be00760-ece6-49c2-97f9-26a0daab728e-istiod-ca-cert" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:24:48.092078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.092050 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.092177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.092083 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be00760-ece6-49c2-97f9-26a0daab728e-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.092177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.092100 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.092177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.092114 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.092177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.092129 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.093149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.093114 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-envoy" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:24:48.093336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.093165 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-istio-token" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:48.093336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.093246 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6be00760-ece6-49c2-97f9-26a0daab728e-istio-podinfo" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:24:48.093521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.093496 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-kube-api-access-jf4w9" pod "6be00760-ece6-49c2-97f9-26a0daab728e" (UID: "6be00760-ece6-49c2-97f9-26a0daab728e"). InnerVolumeSpecName "kube-api-access-jf4w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:24:48.193953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.193921 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.193953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.193950 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be00760-ece6-49c2-97f9-26a0daab728e-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.194133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.193961 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4w9\" (UniqueName: \"kubernetes.io/projected/6be00760-ece6-49c2-97f9-26a0daab728e-kube-api-access-jf4w9\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.194133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.193972 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be00760-ece6-49c2-97f9-26a0daab728e-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:24:48.747929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.747895 5824 generic.go:356] "Generic (PLEG): container finished" podID="6be00760-ece6-49c2-97f9-26a0daab728e" containerID="b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e" exitCode=0 Oct 30 14:24:48.748415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.747951 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" event={"ID":"6be00760-ece6-49c2-97f9-26a0daab728e","Type":"ContainerDied","Data":"b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e"} Oct 30 14:24:48.748415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.747969 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" Oct 30 14:24:48.748415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.747991 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl" event={"ID":"6be00760-ece6-49c2-97f9-26a0daab728e","Type":"ContainerDied","Data":"595b5d77397728315150577411fb1eb0c5b7d885a0dc18dd1fbb3cc8275818c6"} Oct 30 14:24:48.748415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.748010 5824 scope.go:122] "RemoveContainer" containerID="b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e" Oct 30 14:24:48.759403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.759383 5824 scope.go:122] "RemoveContainer" containerID="b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e" Oct 30 14:24:48.759682 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:24:48.759658 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e\": container with ID starting with b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e not found: ID does not exist" containerID="b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e" Oct 30 14:24:48.759778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.759690 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e"} err="failed to get container status \"b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e\": rpc error: code = NotFound desc = could not find container \"b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e\": container with ID starting with b9c994ff8f7eb9d0e825542836afbb86b3d7feed85ab7e28199726989f1fbc9e not found: ID does not exist" Oct 30 14:24:48.824345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.824320 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl"] Oct 30 14:24:48.838369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:48.838336 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-g6hjl"] Oct 30 14:24:50.324488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:24:50.324442 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" path="/var/lib/kubelet/pods/6be00760-ece6-49c2-97f9-26a0daab728e/volumes" Oct 30 14:25:02.870628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:02.870593 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf"] Oct 30 14:25:02.873423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:02.873393 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" Oct 30 14:25:02.873608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:02.873593 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" Oct 30 14:25:02.874127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:02.874107 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be00760-ece6-49c2-97f9-26a0daab728e" containerName="istio-proxy" Oct 30 14:25:02.880171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:02.880151 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:03.012214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:02.960457 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-93549e\"" Oct 30 14:25:03.061216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.006454 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-93549e-dockercfg-l4kgf\"" Oct 30 14:25:03.061216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.007941 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf"] Oct 30 14:25:03.061216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.018006 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkgp\" (UniqueName: \"kubernetes.io/projected/6a948cff-ab50-424c-bc53-b5d16ff54c84-kube-api-access-9pkgp\") pod \"payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf\" (UID: \"6a948cff-ab50-424c-bc53-b5d16ff54c84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:03.062360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.062327 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a948cff-ab50-424c-bc53-b5d16ff54c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf\" (UID: \"6a948cff-ab50-424c-bc53-b5d16ff54c84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:03.164054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.163971 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkgp\" (UniqueName: \"kubernetes.io/projected/6a948cff-ab50-424c-bc53-b5d16ff54c84-kube-api-access-9pkgp\") pod \"payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf\" (UID: \"6a948cff-ab50-424c-bc53-b5d16ff54c84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:03.164301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.164281 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a948cff-ab50-424c-bc53-b5d16ff54c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf\" (UID: \"6a948cff-ab50-424c-bc53-b5d16ff54c84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:03.164934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.164913 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a948cff-ab50-424c-bc53-b5d16ff54c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf\" (UID: \"6a948cff-ab50-424c-bc53-b5d16ff54c84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:03.190999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.190977 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkgp\" (UniqueName: \"kubernetes.io/projected/6a948cff-ab50-424c-bc53-b5d16ff54c84-kube-api-access-9pkgp\") pod \"payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf\" (UID: \"6a948cff-ab50-424c-bc53-b5d16ff54c84\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:03.345322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.345298 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:03.702622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.702497 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf"] Oct 30 14:25:03.797113 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:25:03.797068 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:25:03.872907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:03.872875 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" event={"ID":"6a948cff-ab50-424c-bc53-b5d16ff54c84","Type":"ContainerStarted","Data":"97fc121077555d6c6c895cd52ae69add56897d3864d5de9d7a3a197200cde964"} Oct 30 14:25:04.883881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:04.883838 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" event={"ID":"6a948cff-ab50-424c-bc53-b5d16ff54c84","Type":"ContainerStarted","Data":"e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d"} Oct 30 14:25:04.884424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:04.883925 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:04.934212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:04.933235 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" podStartSLOduration=2.933214916 podStartE2EDuration="2.933214916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:04.929270058 +0000 UTC m=+2134.729775503" watchObservedRunningTime="2026-10-30 14:25:04.933214916 +0000 UTC m=+2134.733720362" Oct 30 14:25:15.900461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:15.900430 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:25:27.695202 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.695157 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n"] Oct 30 14:25:27.699993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.699972 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:27.717921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.717894 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n"] Oct 30 14:25:27.727508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.727486 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0b1c15-dockercfg-hf48x\"" Oct 30 14:25:27.728750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.728724 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0b1c15\"" Oct 30 14:25:27.836211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.821563 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwr7s\" (UniqueName: \"kubernetes.io/projected/0fcdf182-ec74-4302-9fbe-1a9068aa263c-kube-api-access-wwr7s\") pod \"payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n\" (UID: \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:27.836211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.821964 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf182-ec74-4302-9fbe-1a9068aa263c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n\" (UID: \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:27.927698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.927658 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf182-ec74-4302-9fbe-1a9068aa263c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n\" (UID: \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:27.928020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.927997 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwr7s\" (UniqueName: \"kubernetes.io/projected/0fcdf182-ec74-4302-9fbe-1a9068aa263c-kube-api-access-wwr7s\") pod \"payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n\" (UID: \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:27.938392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.938336 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf182-ec74-4302-9fbe-1a9068aa263c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n\" (UID: \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:27.958340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.958249 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65"] Oct 30 14:25:27.980354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.980317 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65"] Oct 30 14:25:27.980584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.980442 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:27.983653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:27.983618 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwr7s\" (UniqueName: \"kubernetes.io/projected/0fcdf182-ec74-4302-9fbe-1a9068aa263c-kube-api-access-wwr7s\") pod \"payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n\" (UID: \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:28.014505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.014483 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:28.030504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.030476 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a064383-9f17-459d-b2d6-0600806167fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65\" (UID: \"5a064383-9f17-459d-b2d6-0600806167fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:28.030633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.030601 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bmj\" (UniqueName: \"kubernetes.io/projected/5a064383-9f17-459d-b2d6-0600806167fd-kube-api-access-v2bmj\") pod \"payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65\" (UID: \"5a064383-9f17-459d-b2d6-0600806167fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:28.131730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.131701 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a064383-9f17-459d-b2d6-0600806167fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65\" (UID: \"5a064383-9f17-459d-b2d6-0600806167fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:28.131909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.131764 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bmj\" (UniqueName: \"kubernetes.io/projected/5a064383-9f17-459d-b2d6-0600806167fd-kube-api-access-v2bmj\") pod \"payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65\" (UID: \"5a064383-9f17-459d-b2d6-0600806167fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:28.132929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.132876 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a064383-9f17-459d-b2d6-0600806167fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65\" (UID: \"5a064383-9f17-459d-b2d6-0600806167fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:28.158473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.158449 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bmj\" (UniqueName: \"kubernetes.io/projected/5a064383-9f17-459d-b2d6-0600806167fd-kube-api-access-v2bmj\") pod \"payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65\" (UID: \"5a064383-9f17-459d-b2d6-0600806167fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:28.348904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.348800 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:28.461170 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.461140 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n"] Oct 30 14:25:28.696660 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:25:28.696624 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a064383_9f17_459d_b2d6_0600806167fd.slice/crio-b621633a13bb21c7b5255c02a4bda6b93957a435be3a429af1a1c35595da3e62 WatchSource:0}: Error finding container b621633a13bb21c7b5255c02a4bda6b93957a435be3a429af1a1c35595da3e62: Status 404 returned error can't find the container with id b621633a13bb21c7b5255c02a4bda6b93957a435be3a429af1a1c35595da3e62 Oct 30 14:25:28.701949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:28.701854 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65"] Oct 30 14:25:29.105559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:29.105466 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" event={"ID":"5a064383-9f17-459d-b2d6-0600806167fd","Type":"ContainerStarted","Data":"ba8418417d19b371a43d712ba6a3ed9bbe06be6d7957e23d12603a0efa0d4ca6"} Oct 30 14:25:29.105559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:29.105531 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" event={"ID":"5a064383-9f17-459d-b2d6-0600806167fd","Type":"ContainerStarted","Data":"b621633a13bb21c7b5255c02a4bda6b93957a435be3a429af1a1c35595da3e62"} Oct 30 14:25:29.105774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:29.105694 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:29.107536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:29.107504 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" event={"ID":"0fcdf182-ec74-4302-9fbe-1a9068aa263c","Type":"ContainerStarted","Data":"a8d094220567b5f84043a2bd53cc5a75bcc78c13dc9ff34749e78464dcc2e506"} Oct 30 14:25:29.107649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:29.107557 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" event={"ID":"0fcdf182-ec74-4302-9fbe-1a9068aa263c","Type":"ContainerStarted","Data":"e74361be8e869314cab9435587dc927359f0d960f82b746ec024eadb0ce2275f"} Oct 30 14:25:29.107649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:29.107579 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:25:29.160839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:29.157822 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" podStartSLOduration=2.157803161 podStartE2EDuration="2.157803161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:29.155809916 +0000 UTC m=+2158.956315361" watchObservedRunningTime="2026-10-30 14:25:29.157803161 +0000 UTC m=+2158.958308606" Oct 30 14:25:29.202139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:29.202061 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" podStartSLOduration=2.202044457 podStartE2EDuration="2.202044457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:25:29.200512213 +0000 UTC m=+2159.001017671" watchObservedRunningTime="2026-10-30 14:25:29.202044457 +0000 UTC m=+2159.002549901" Oct 30 14:25:33.646408 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:25:33.646371 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:25:40.118540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:40.118512 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:25:40.118950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:25:40.118583 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:26:03.645902 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:26:03.645856 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:26:08.386869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.386831 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7"] Oct 30 14:26:08.395211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.392849 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.415907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.415877 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7"] Oct 30 14:26:08.476526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.476492 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lqg\" (UniqueName: \"kubernetes.io/projected/550a4ed6-6139-4011-add0-4591fe001902-kube-api-access-s7lqg\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.476729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.476657 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a4ed6-6139-4011-add0-4591fe001902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.476729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.476702 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a4ed6-6139-4011-add0-4591fe001902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.478800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.478771 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:26:08.479176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.479154 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:26:08.578112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.578072 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lqg\" (UniqueName: \"kubernetes.io/projected/550a4ed6-6139-4011-add0-4591fe001902-kube-api-access-s7lqg\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.578330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.578162 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a4ed6-6139-4011-add0-4591fe001902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.578330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.578205 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a4ed6-6139-4011-add0-4591fe001902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.578974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.578948 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a4ed6-6139-4011-add0-4591fe001902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.580909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.580874 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a4ed6-6139-4011-add0-4591fe001902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.617273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.617240 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lqg\" (UniqueName: \"kubernetes.io/projected/550a4ed6-6139-4011-add0-4591fe001902-kube-api-access-s7lqg\") pod \"maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:08.709340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:08.709301 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:09.028219 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:26:09.028176 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550a4ed6_6139_4011_add0_4591fe001902.slice/crio-7a7e5f08a8e31f254292acdcdca3869542380419436d4fc4ee68f098a18c1194 WatchSource:0}: Error finding container 7a7e5f08a8e31f254292acdcdca3869542380419436d4fc4ee68f098a18c1194: Status 404 returned error can't find the container with id 7a7e5f08a8e31f254292acdcdca3869542380419436d4fc4ee68f098a18c1194 Oct 30 14:26:09.029236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:09.029206 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7"] Oct 30 14:26:09.484942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:09.484908 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" event={"ID":"550a4ed6-6139-4011-add0-4591fe001902","Type":"ContainerStarted","Data":"87684eba665e8c925437ce7b5438a3c8c87e00db3a3f9cb6b54689d8fb18c0ac"} Oct 30 14:26:09.485326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:09.484972 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" event={"ID":"550a4ed6-6139-4011-add0-4591fe001902","Type":"ContainerStarted","Data":"7a7e5f08a8e31f254292acdcdca3869542380419436d4fc4ee68f098a18c1194"} Oct 30 14:26:09.529836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:09.529794 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" podStartSLOduration=1.529780111 podStartE2EDuration="1.529780111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:09.528739887 +0000 UTC m=+2199.329245330" watchObservedRunningTime="2026-10-30 14:26:09.529780111 +0000 UTC m=+2199.330285554" Oct 30 14:26:10.496318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:10.496285 5824 generic.go:356] "Generic (PLEG): container finished" podID="550a4ed6-6139-4011-add0-4591fe001902" containerID="87684eba665e8c925437ce7b5438a3c8c87e00db3a3f9cb6b54689d8fb18c0ac" exitCode=0 Oct 30 14:26:10.496709 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:10.496365 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" event={"ID":"550a4ed6-6139-4011-add0-4591fe001902","Type":"ContainerDied","Data":"87684eba665e8c925437ce7b5438a3c8c87e00db3a3f9cb6b54689d8fb18c0ac"} Oct 30 14:26:11.703152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.703126 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:11.840717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.840633 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/550a4ed6-6139-4011-add0-4591fe001902-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a4ed6-6139-4011-add0-4591fe001902-maas-api-xks-ca\") pod \"550a4ed6-6139-4011-add0-4591fe001902\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " Oct 30 14:26:11.840717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.840691 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550a4ed6-6139-4011-add0-4591fe001902-kube-api-access-s7lqg\" (UniqueName: \"kubernetes.io/projected/550a4ed6-6139-4011-add0-4591fe001902-kube-api-access-s7lqg\") pod \"550a4ed6-6139-4011-add0-4591fe001902\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " Oct 30 14:26:11.840980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.840966 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550a4ed6-6139-4011-add0-4591fe001902-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a4ed6-6139-4011-add0-4591fe001902-maas-api-openshift-ca\") pod \"550a4ed6-6139-4011-add0-4591fe001902\" (UID: \"550a4ed6-6139-4011-add0-4591fe001902\") " Oct 30 14:26:11.841340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.841309 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550a4ed6-6139-4011-add0-4591fe001902-maas-api-openshift-ca" pod "550a4ed6-6139-4011-add0-4591fe001902" (UID: "550a4ed6-6139-4011-add0-4591fe001902"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:11.841540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.841524 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a4ed6-6139-4011-add0-4591fe001902-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:11.842769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.842739 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550a4ed6-6139-4011-add0-4591fe001902-maas-api-xks-ca" pod "550a4ed6-6139-4011-add0-4591fe001902" (UID: "550a4ed6-6139-4011-add0-4591fe001902"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:26:11.842884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.842826 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550a4ed6-6139-4011-add0-4591fe001902-kube-api-access-s7lqg" pod "550a4ed6-6139-4011-add0-4591fe001902" (UID: "550a4ed6-6139-4011-add0-4591fe001902"). InnerVolumeSpecName "kube-api-access-s7lqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:11.943321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.943289 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a4ed6-6139-4011-add0-4591fe001902-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:11.943321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:11.943318 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7lqg\" (UniqueName: \"kubernetes.io/projected/550a4ed6-6139-4011-add0-4591fe001902-kube-api-access-s7lqg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:12.513094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:12.513065 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" Oct 30 14:26:12.513300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:12.513093 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7" event={"ID":"550a4ed6-6139-4011-add0-4591fe001902","Type":"ContainerDied","Data":"7a7e5f08a8e31f254292acdcdca3869542380419436d4fc4ee68f098a18c1194"} Oct 30 14:26:12.513300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:12.513137 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a7e5f08a8e31f254292acdcdca3869542380419436d4fc4ee68f098a18c1194" Oct 30 14:26:14.596954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:14.596923 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt"] Oct 30 14:26:14.597767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:14.597733 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" containerID="cri-o://104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c" gracePeriod=30 Oct 30 14:26:14.608500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:14.608473 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds"] Oct 30 14:26:14.608717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:14.608694 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" containerID="cri-o://88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5" gracePeriod=30 Oct 30 14:26:14.614074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:14.614052 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8"] Oct 30 14:26:14.614295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:14.614266 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" containerID="cri-o://e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86" gracePeriod=30 Oct 30 14:26:14.892065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:14.891847 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:14.892065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:14.891906 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:15.135013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.134975 5824 patch_prober.go:29] interesting pod/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:15.135181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.135036 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:15.939287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.939256 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc"] Oct 30 14:26:15.940082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.940065 5824 cpu_manager.go:414] "containerMap: removing container" podUID="550a4ed6-6139-4011-add0-4591fe001902" containerName="revoke-keys" Oct 30 14:26:15.940082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.940082 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550a4ed6-6139-4011-add0-4591fe001902" containerName="revoke-keys" Oct 30 14:26:15.940247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.940236 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="550a4ed6-6139-4011-add0-4591fe001902" containerName="revoke-keys" Oct 30 14:26:15.944483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.944465 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:15.962655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.962632 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-019e5b-openshift-default-dockercfg-ls2fm\"" Oct 30 14:26:15.990717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:15.990695 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc"] Oct 30 14:26:16.011062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011036 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-token\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.011243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011221 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-data\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.011337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011262 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-podinfo\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.011388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011373 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-credential-socket\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.011436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011415 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-socket\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.011481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011449 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istiod-ca-cert\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.011481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011466 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn7sl\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-kube-api-access-nn7sl\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.011592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011576 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-certs\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.011627 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.011618 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-envoy\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.113509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.113469 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-data\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.113827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.113515 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-podinfo\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.113827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.113605 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-credential-socket\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.113827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.113649 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-socket\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.113827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.113693 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istiod-ca-cert\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.113827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.113710 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn7sl\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-kube-api-access-nn7sl\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.113827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.113807 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-certs\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.114166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.113874 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-envoy\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.114166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.114044 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-token\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.114166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.114101 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-credential-socket\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.114360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.114205 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-data\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.114360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.114281 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-certs\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.114533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.114514 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istiod-ca-cert\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.114600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.114538 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-socket\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.116406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.116383 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-envoy\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.116501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.116450 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-podinfo\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.161252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.161224 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn7sl\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-kube-api-access-nn7sl\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.161454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.161436 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-token\") pod \"e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.259835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.259760 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:16.618767 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:26:16.618737 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05b8bdf_8389_4448_b6fa_2b995ff94f36.slice/crio-42deadc0df850193ef0d9896eb244e7d9324c2a9db0fb395f767889a40cb6ad7 WatchSource:0}: Error finding container 42deadc0df850193ef0d9896eb244e7d9324c2a9db0fb395f767889a40cb6ad7: Status 404 returned error can't find the container with id 42deadc0df850193ef0d9896eb244e7d9324c2a9db0fb395f767889a40cb6ad7 Oct 30 14:26:16.620973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.620949 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc"] Oct 30 14:26:16.621764 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.621731 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:16.621817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.621800 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:16.621850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:16.621830 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:26:17.585213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:17.581361 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" event={"ID":"f05b8bdf-8389-4448-b6fa-2b995ff94f36","Type":"ContainerStarted","Data":"306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8"} Oct 30 14:26:17.585213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:17.581447 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" event={"ID":"f05b8bdf-8389-4448-b6fa-2b995ff94f36","Type":"ContainerStarted","Data":"42deadc0df850193ef0d9896eb244e7d9324c2a9db0fb395f767889a40cb6ad7"} Oct 30 14:26:17.653963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:17.653914 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" podStartSLOduration=2.653900077 podStartE2EDuration="2.653900077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:17.650592435 +0000 UTC m=+2207.451097880" watchObservedRunningTime="2026-10-30 14:26:17.653900077 +0000 UTC m=+2207.454405521" Oct 30 14:26:18.260958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:18.260912 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:18.273029 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:18.272993 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:18.590352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:18.590269 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:18.591392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:18.591369 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:26:18.759614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:18.759573 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx"] Oct 30 14:26:19.931824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.931789 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:26:19.973401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.973377 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:26:19.986485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.986461 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:26:19.990666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.990645 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-envoy\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.990756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.990746 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efac379e-c863-4eac-af03-5a727f03f70a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac379e-c863-4eac-af03-5a727f03f70a-istiod-ca-cert\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.990793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.990772 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-credential-socket\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.990828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.990806 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-kube-api-access-7tgzk\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-kube-api-access-7tgzk\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.990872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.990842 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-data\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.990912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.990880 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-credential-socket\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.990967 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.990959 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efac379e-c863-4eac-af03-5a727f03f70a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac379e-c863-4eac-af03-5a727f03f70a-istio-podinfo\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.991013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.990981 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-podinfo\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.991064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991021 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-socket\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.991135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991089 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-token\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-token\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.991224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991140 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-certs\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.991224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991217 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-istio-token\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-istio-token\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.991341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991235 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-data" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.991341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991245 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efac379e-c863-4eac-af03-5a727f03f70a-istiod-ca-cert" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:19.991341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991264 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-credential-socket" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.991341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991267 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-envoy\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.991569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991356 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-socket\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.991569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991397 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-credential-socket" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.991569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991457 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-certs\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.991569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991491 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-data\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.991569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991521 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-kube-api-access-4h9bn\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-kube-api-access-4h9bn\") pod \"efac379e-c863-4eac-af03-5a727f03f70a\" (UID: \"efac379e-c863-4eac-af03-5a727f03f70a\") " Oct 30 14:26:19.991758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991608 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-socket" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.991758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991666 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cdd0b83-39b9-4b30-89ba-337004b917ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd0b83-39b9-4b30-89ba-337004b917ea-istiod-ca-cert\") pod \"7cdd0b83-39b9-4b30-89ba-337004b917ea\" (UID: \"7cdd0b83-39b9-4b30-89ba-337004b917ea\") " Oct 30 14:26:19.991821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991803 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-certs" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.992011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.991992 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-socket" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.992387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.992361 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdd0b83-39b9-4b30-89ba-337004b917ea-istiod-ca-cert" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:19.992685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.992660 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-data" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993476 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993505 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993521 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993536 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993549 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd0b83-39b9-4b30-89ba-337004b917ea-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993563 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac379e-c863-4eac-af03-5a727f03f70a-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993576 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993591 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993605 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:19.996367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.993571 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-certs" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.998490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.998105 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-envoy" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.998581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.998502 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-envoy" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:19.998646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.998586 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-podinfo" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:19.998700 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.998654 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-istio-token" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:19.998996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.998969 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-token" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:19.999089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.999053 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-kube-api-access-7tgzk" pod "7cdd0b83-39b9-4b30-89ba-337004b917ea" (UID: "7cdd0b83-39b9-4b30-89ba-337004b917ea"). InnerVolumeSpecName "kube-api-access-7tgzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:19.999294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.999149 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efac379e-c863-4eac-af03-5a727f03f70a-istio-podinfo" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:19.999474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:19.999435 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-kube-api-access-4h9bn" pod "efac379e-c863-4eac-af03-5a727f03f70a" (UID: "efac379e-c863-4eac-af03-5a727f03f70a"). InnerVolumeSpecName "kube-api-access-4h9bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.094757 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-certs\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.094849 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-kube-api-access-djf5l\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-kube-api-access-djf5l\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.094911 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-envoy\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.095032 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ae12b0-3781-48e3-a967-13e946c58fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ae12b0-3781-48e3-a967-13e946c58fc1-istiod-ca-cert\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.095084 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-certs" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.095143 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-podinfo\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.095264 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-data\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.095306 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-socket\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.095374 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-token\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.095432 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-credential-socket\") pod \"16ae12b0-3781-48e3-a967-13e946c58fc1\" (UID: \"16ae12b0-3781-48e3-a967-13e946c58fc1\") " Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.096141 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ae12b0-3781-48e3-a967-13e946c58fc1-istiod-ca-cert" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097053 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac379e-c863-4eac-af03-5a727f03f70a-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097073 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097082 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097091 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097100 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097108 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097116 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac379e-c863-4eac-af03-5a727f03f70a-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097124 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9bn\" (UniqueName: \"kubernetes.io/projected/efac379e-c863-4eac-af03-5a727f03f70a-kube-api-access-4h9bn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097135 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16ae12b0-3781-48e3-a967-13e946c58fc1-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097143 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cdd0b83-39b9-4b30-89ba-337004b917ea-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097151 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tgzk\" (UniqueName: \"kubernetes.io/projected/7cdd0b83-39b9-4b30-89ba-337004b917ea-kube-api-access-7tgzk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.097326 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-socket" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.099129 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-data" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.104239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.099342 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-credential-socket" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.105828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.105519 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-podinfo" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:20.105828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.105608 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-envoy" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:20.105828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.105762 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-kube-api-access-djf5l" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "kube-api-access-djf5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.107342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.107314 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-token" pod "16ae12b0-3781-48e3-a967-13e946c58fc1" (UID: "16ae12b0-3781-48e3-a967-13e946c58fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:20.199434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.199390 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.199434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.199437 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.199719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.199456 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djf5l\" (UniqueName: \"kubernetes.io/projected/16ae12b0-3781-48e3-a967-13e946c58fc1-kube-api-access-djf5l\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.199719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.199473 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.199719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.199486 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.199719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.199499 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.199719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.199513 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16ae12b0-3781-48e3-a967-13e946c58fc1-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:20.606913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.606825 5824 generic.go:356] "Generic (PLEG): container finished" podID="efac379e-c863-4eac-af03-5a727f03f70a" containerID="104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c" exitCode=0 Oct 30 14:26:20.606913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.606875 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" Oct 30 14:26:20.607169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.606911 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" event={"ID":"efac379e-c863-4eac-af03-5a727f03f70a","Type":"ContainerDied","Data":"104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c"} Oct 30 14:26:20.607169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.606959 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt" event={"ID":"efac379e-c863-4eac-af03-5a727f03f70a","Type":"ContainerDied","Data":"a71cb3b6f3b66e59146f9e487e8cef259ffe000e1d56c513ac46cda3f468e22d"} Oct 30 14:26:20.607169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.606980 5824 scope.go:122] "RemoveContainer" containerID="104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c" Oct 30 14:26:20.608544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.608521 5824 generic.go:356] "Generic (PLEG): container finished" podID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerID="e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86" exitCode=0 Oct 30 14:26:20.608658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.608552 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" event={"ID":"16ae12b0-3781-48e3-a967-13e946c58fc1","Type":"ContainerDied","Data":"e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86"} Oct 30 14:26:20.608658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.608584 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" Oct 30 14:26:20.608658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.608595 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8" event={"ID":"16ae12b0-3781-48e3-a967-13e946c58fc1","Type":"ContainerDied","Data":"2b660990df8e4fdfaeb1db5cb4e1f6080f60242fdb837b192e64691a90bc56f1"} Oct 30 14:26:20.610405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.610380 5824 generic.go:356] "Generic (PLEG): container finished" podID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerID="88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5" exitCode=0 Oct 30 14:26:20.610526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.610414 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" event={"ID":"7cdd0b83-39b9-4b30-89ba-337004b917ea","Type":"ContainerDied","Data":"88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5"} Oct 30 14:26:20.610526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.610443 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" Oct 30 14:26:20.610526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.610459 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds" event={"ID":"7cdd0b83-39b9-4b30-89ba-337004b917ea","Type":"ContainerDied","Data":"654eef8cbaf8c95f3a25cae34b28da0f00f9e5c8003ff855697c243e6fed1166"} Oct 30 14:26:20.620849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.620820 5824 scope.go:122] "RemoveContainer" containerID="104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c" Oct 30 14:26:20.621152 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:26:20.621134 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c\": container with ID starting with 104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c not found: ID does not exist" containerID="104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c" Oct 30 14:26:20.621227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.621158 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c"} err="failed to get container status \"104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c\": rpc error: code = NotFound desc = could not find container \"104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c\": container with ID starting with 104febeef7808687da40791324cfb268b533251aa61fbf60f15ad0544c057a1c not found: ID does not exist" Oct 30 14:26:20.621227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.621174 5824 scope.go:122] "RemoveContainer" containerID="e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86" Oct 30 14:26:20.649889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.649866 5824 scope.go:122] "RemoveContainer" containerID="e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86" Oct 30 14:26:20.650287 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:26:20.650259 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86\": container with ID starting with e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86 not found: ID does not exist" containerID="e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86" Oct 30 14:26:20.650427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.650298 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86"} err="failed to get container status \"e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86\": rpc error: code = NotFound desc = could not find container \"e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86\": container with ID starting with e8e11d65f874e62f0b8137c6d8fd7c254eeb5bfd4f3710234446c6b84f1fdd86 not found: ID does not exist" Oct 30 14:26:20.650427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.650322 5824 scope.go:122] "RemoveContainer" containerID="88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5" Oct 30 14:26:20.661569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.661550 5824 scope.go:122] "RemoveContainer" containerID="88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5" Oct 30 14:26:20.661860 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:26:20.661838 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5\": container with ID starting with 88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5 not found: ID does not exist" containerID="88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5" Oct 30 14:26:20.661931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.661871 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5"} err="failed to get container status \"88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5\": rpc error: code = NotFound desc = could not find container \"88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5\": container with ID starting with 88d1546b1a11f3045f1d10beaa82537238fe9c4dfcc4a02bc493ed0fdc3db1b5 not found: ID does not exist" Oct 30 14:26:20.670092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.670072 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt"] Oct 30 14:26:20.683293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.683268 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-4rgtt"] Oct 30 14:26:20.723256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.723234 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds"] Oct 30 14:26:20.731281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.731255 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-9crds"] Oct 30 14:26:20.772319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.772282 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8"] Oct 30 14:26:20.775458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:20.775438 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea9ed2-openshift-default-6b9bddc5c9-djct8"] Oct 30 14:26:22.319348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:22.319306 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" path="/var/lib/kubelet/pods/16ae12b0-3781-48e3-a967-13e946c58fc1/volumes" Oct 30 14:26:22.320001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:22.319978 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" path="/var/lib/kubelet/pods/7cdd0b83-39b9-4b30-89ba-337004b917ea/volumes" Oct 30 14:26:22.320537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:22.320515 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efac379e-c863-4eac-af03-5a727f03f70a" path="/var/lib/kubelet/pods/efac379e-c863-4eac-af03-5a727f03f70a/volumes" Oct 30 14:26:23.760477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:23.760445 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" podUID="65da2b41-a7a1-47b2-89ce-424fba8b82a5" containerName="payload-processing" containerID="cri-o://0a1f1f5a7d64419a86e8c53ce93fce0f437dc89dc61ccbced62cd1e2445a2207" gracePeriod=25 Oct 30 14:26:24.963548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:24.963511 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" podUID="65da2b41-a7a1-47b2-89ce-424fba8b82a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:30.201698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:30.201666 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7"] Oct 30 14:26:30.214602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:30.214576 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a9d7b-j9xw7"] Oct 30 14:26:30.334181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:30.334146 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550a4ed6-6139-4011-add0-4591fe001902" path="/var/lib/kubelet/pods/550a4ed6-6139-4011-add0-4591fe001902/volumes" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.643557 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl"] Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645076 5824 cpu_manager.go:414] "containerMap: removing container" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645096 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645122 5824 cpu_manager.go:414] "containerMap: removing container" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645130 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645158 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645167 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645382 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ae12b0-3781-48e3-a967-13e946c58fc1" containerName="istio-proxy" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645401 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="efac379e-c863-4eac-af03-5a727f03f70a" containerName="istio-proxy" Oct 30 14:26:32.657212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.645410 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdd0b83-39b9-4b30-89ba-337004b917ea" containerName="istio-proxy" Oct 30 14:26:32.663332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.662389 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:32.722531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.702443 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-019e5b\"" Oct 30 14:26:32.722531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.702597 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-019e5b-dockercfg-r6m5t\"" Oct 30 14:26:32.771340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.760788 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl"] Oct 30 14:26:32.801211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.799891 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qwx\" (UniqueName: \"kubernetes.io/projected/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-kube-api-access-w2qwx\") pod \"payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl\" (UID: \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:32.801211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.800044 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl\" (UID: \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:32.901531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.901495 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl\" (UID: \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:32.901830 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.901808 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qwx\" (UniqueName: \"kubernetes.io/projected/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-kube-api-access-w2qwx\") pod \"payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl\" (UID: \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:32.902202 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.902173 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl\" (UID: \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:32.928658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:32.928628 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qwx\" (UniqueName: \"kubernetes.io/projected/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-kube-api-access-w2qwx\") pod \"payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl\" (UID: \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:33.158917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:33.158884 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:33.468140 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:26:33.468079 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bebc04_7fc4_45a4_9468_4fd4d7f7a2d9.slice/crio-0da0956edb18c1026175d5dc83f2f28f9e0e840edbb21b26c2107303e4756678 WatchSource:0}: Error finding container 0da0956edb18c1026175d5dc83f2f28f9e0e840edbb21b26c2107303e4756678: Status 404 returned error can't find the container with id 0da0956edb18c1026175d5dc83f2f28f9e0e840edbb21b26c2107303e4756678 Oct 30 14:26:33.469228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:33.469202 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl"] Oct 30 14:26:33.641322 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:26:33.641287 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:26:33.862126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:33.862038 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" event={"ID":"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9","Type":"ContainerStarted","Data":"ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041"} Oct 30 14:26:33.862126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:33.862096 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" event={"ID":"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9","Type":"ContainerStarted","Data":"0da0956edb18c1026175d5dc83f2f28f9e0e840edbb21b26c2107303e4756678"} Oct 30 14:26:33.862561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:33.862132 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:33.929844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:33.929795 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" podStartSLOduration=1.929781797 podStartE2EDuration="1.929781797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:26:33.927881922 +0000 UTC m=+2223.728387391" watchObservedRunningTime="2026-10-30 14:26:33.929781797 +0000 UTC m=+2223.730287240" Oct 30 14:26:34.964321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:34.964283 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" podUID="65da2b41-a7a1-47b2-89ce-424fba8b82a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:26:43.641900 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:43.641864 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25"] Oct 30 14:26:43.642492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:43.642094 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" containerID="cri-o://05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573" gracePeriod=30 Oct 30 14:26:43.973623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:43.973583 5824 generic.go:356] "Generic (PLEG): container finished" podID="65da2b41-a7a1-47b2-89ce-424fba8b82a5" containerID="0a1f1f5a7d64419a86e8c53ce93fce0f437dc89dc61ccbced62cd1e2445a2207" exitCode=0 Oct 30 14:26:43.973623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:43.973665 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" event={"ID":"65da2b41-a7a1-47b2-89ce-424fba8b82a5","Type":"ContainerDied","Data":"0a1f1f5a7d64419a86e8c53ce93fce0f437dc89dc61ccbced62cd1e2445a2207"} Oct 30 14:26:44.085993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.085970 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:26:44.184719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.184676 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65da2b41-a7a1-47b2-89ce-424fba8b82a5-kube-api-access-x5g8f\" (UniqueName: \"kubernetes.io/projected/65da2b41-a7a1-47b2-89ce-424fba8b82a5-kube-api-access-x5g8f\") pod \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\" (UID: \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\") " Oct 30 14:26:44.184901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.184751 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65da2b41-a7a1-47b2-89ce-424fba8b82a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65da2b41-a7a1-47b2-89ce-424fba8b82a5-plugins-config-volume\") pod \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\" (UID: \"65da2b41-a7a1-47b2-89ce-424fba8b82a5\") " Oct 30 14:26:44.185219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.185158 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65da2b41-a7a1-47b2-89ce-424fba8b82a5-plugins-config-volume" pod "65da2b41-a7a1-47b2-89ce-424fba8b82a5" (UID: "65da2b41-a7a1-47b2-89ce-424fba8b82a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:44.185631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.185614 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65da2b41-a7a1-47b2-89ce-424fba8b82a5-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:44.187519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.187489 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65da2b41-a7a1-47b2-89ce-424fba8b82a5-kube-api-access-x5g8f" pod "65da2b41-a7a1-47b2-89ce-424fba8b82a5" (UID: "65da2b41-a7a1-47b2-89ce-424fba8b82a5"). InnerVolumeSpecName "kube-api-access-x5g8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:44.287638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.287606 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5g8f\" (UniqueName: \"kubernetes.io/projected/65da2b41-a7a1-47b2-89ce-424fba8b82a5-kube-api-access-x5g8f\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:44.874308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.874275 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:26:44.987127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.987087 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" Oct 30 14:26:44.987349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.987094 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx" event={"ID":"65da2b41-a7a1-47b2-89ce-424fba8b82a5","Type":"ContainerDied","Data":"55d46015db325e7290966e1e7c72b40e5f05ec8190dc1cd85c583d8fcad26aa1"} Oct 30 14:26:44.987349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:44.987265 5824 scope.go:122] "RemoveContainer" containerID="0a1f1f5a7d64419a86e8c53ce93fce0f437dc89dc61ccbced62cd1e2445a2207" Oct 30 14:26:45.047408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:45.047375 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx"] Oct 30 14:26:45.052787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:45.052761 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5a9d7b-6f7494c4b8-flvrx"] Oct 30 14:26:46.319492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:46.319462 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65da2b41-a7a1-47b2-89ce-424fba8b82a5" path="/var/lib/kubelet/pods/65da2b41-a7a1-47b2-89ce-424fba8b82a5/volumes" Oct 30 14:26:48.228431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.228393 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:26:48.228795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.228451 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:26:48.940058 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.940032 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:26:48.964481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964454 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-data\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.964586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964504 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-certs\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.964641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964624 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-istio-token\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-istio-token\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.964703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964687 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629544b6-29fe-475f-8833-bca9cca08552-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629544b6-29fe-475f-8833-bca9cca08552-istiod-ca-cert\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.964745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964728 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-credential-socket\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.964793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964687 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-data" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.964840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964818 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-certs" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.964894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964851 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-socket\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.964953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964902 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-envoy\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.964953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964942 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/629544b6-29fe-475f-8833-bca9cca08552-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629544b6-29fe-475f-8833-bca9cca08552-istio-podinfo\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.965061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.964996 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-kube-api-access-gjqtn\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-kube-api-access-gjqtn\") pod \"629544b6-29fe-475f-8833-bca9cca08552\" (UID: \"629544b6-29fe-475f-8833-bca9cca08552\") " Oct 30 14:26:48.965119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.965082 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-credential-socket" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.965119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.965095 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629544b6-29fe-475f-8833-bca9cca08552-istiod-ca-cert" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:26:48.965119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.965101 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-socket" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.965836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.965811 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.965932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.965844 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.965932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.965859 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.965932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.965872 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629544b6-29fe-475f-8833-bca9cca08552-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.965932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.965886 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:48.967010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.966983 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/629544b6-29fe-475f-8833-bca9cca08552-istio-podinfo" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:26:48.967116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.967075 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-kube-api-access-gjqtn" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "kube-api-access-gjqtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:48.967116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.967088 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-envoy" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:26:48.967116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:48.967095 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-istio-token" pod "629544b6-29fe-475f-8833-bca9cca08552" (UID: "629544b6-29fe-475f-8833-bca9cca08552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:26:49.021432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.021397 5824 generic.go:356] "Generic (PLEG): container finished" podID="629544b6-29fe-475f-8833-bca9cca08552" containerID="05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573" exitCode=0 Oct 30 14:26:49.021594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.021463 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" Oct 30 14:26:49.021594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.021481 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" event={"ID":"629544b6-29fe-475f-8833-bca9cca08552","Type":"ContainerDied","Data":"05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573"} Oct 30 14:26:49.021594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.021572 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25" event={"ID":"629544b6-29fe-475f-8833-bca9cca08552","Type":"ContainerDied","Data":"76f14cfe045de65565f1803cca893a93e584fe6d2379d37d190c6a2ace42ba16"} Oct 30 14:26:49.021776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.021603 5824 scope.go:122] "RemoveContainer" containerID="05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573" Oct 30 14:26:49.034695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.034670 5824 scope.go:122] "RemoveContainer" containerID="05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573" Oct 30 14:26:49.034999 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:26:49.034977 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573\": container with ID starting with 05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573 not found: ID does not exist" containerID="05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573" Oct 30 14:26:49.035084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.035005 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573"} err="failed to get container status \"05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573\": rpc error: code = NotFound desc = could not find container \"05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573\": container with ID starting with 05cb45518d161104b0130ce6a373fe6bf9e91b778a197c5e9898e779d8556573 not found: ID does not exist" Oct 30 14:26:49.067698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.067671 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:49.067698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.067693 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629544b6-29fe-475f-8833-bca9cca08552-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:49.067698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.067702 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629544b6-29fe-475f-8833-bca9cca08552-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:49.067882 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.067712 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqtn\" (UniqueName: \"kubernetes.io/projected/629544b6-29fe-475f-8833-bca9cca08552-kube-api-access-gjqtn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:26:49.087765 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.087742 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25"] Oct 30 14:26:49.092925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.092899 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-7qc25"] Oct 30 14:26:49.422459 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:49.422420 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h"] Oct 30 14:26:50.321725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:50.321686 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629544b6-29fe-475f-8833-bca9cca08552" path="/var/lib/kubelet/pods/629544b6-29fe-475f-8833-bca9cca08552/volumes" Oct 30 14:26:54.423008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:54.422951 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" podUID="d2df3c67-41d4-457c-8115-c6449e3d5fe8" containerName="payload-pre-processing" containerID="cri-o://9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6" gracePeriod=25 Oct 30 14:26:58.898762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:58.898729 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6"] Oct 30 14:26:58.899285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:26:58.898955 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" containerID="cri-o://e6e4a8820f531de05088f349cde66b8629319c4b91110e0bfe28e549e62c01fb" gracePeriod=30 Oct 30 14:27:02.072213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:02.072144 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" podUID="d2df3c67-41d4-457c-8115-c6449e3d5fe8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:27:03.641594 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:27:03.641559 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:27:04.140664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.140633 5824 generic.go:356] "Generic (PLEG): container finished" podID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerID="e6e4a8820f531de05088f349cde66b8629319c4b91110e0bfe28e549e62c01fb" exitCode=0 Oct 30 14:27:04.140823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.140706 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" event={"ID":"db815bfe-85a0-48fb-858f-015cceef6ea6","Type":"ContainerDied","Data":"e6e4a8820f531de05088f349cde66b8629319c4b91110e0bfe28e549e62c01fb"} Oct 30 14:27:04.187093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.187071 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:27:04.350232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350181 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-socket\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350234 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-podinfo\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350283 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-kube-api-access-k7gtx\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-kube-api-access-k7gtx\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350376 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-socket" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:04.350468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350429 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-credential-socket\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350458 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-envoy\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350479 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db815bfe-85a0-48fb-858f-015cceef6ea6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db815bfe-85a0-48fb-858f-015cceef6ea6-istiod-ca-cert\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350500 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-certs\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350534 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-token\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-token\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350692 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-data\") pod \"db815bfe-85a0-48fb-858f-015cceef6ea6\" (UID: \"db815bfe-85a0-48fb-858f-015cceef6ea6\") " Oct 30 14:27:04.350801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350743 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-credential-socket" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:04.350915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350853 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-certs" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:04.350915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.350874 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db815bfe-85a0-48fb-858f-015cceef6ea6-istiod-ca-cert" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:04.351247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.351222 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-data" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:04.351410 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.351394 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:04.351478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.351413 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db815bfe-85a0-48fb-858f-015cceef6ea6-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:04.351478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.351423 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:04.351478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.351431 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:04.351478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.351439 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:04.352614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.352586 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-envoy" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:04.352940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.352918 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-token" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:04.353011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.352950 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-podinfo" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:27:04.353047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.353012 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-kube-api-access-k7gtx" pod "db815bfe-85a0-48fb-858f-015cceef6ea6" (UID: "db815bfe-85a0-48fb-858f-015cceef6ea6"). InnerVolumeSpecName "kube-api-access-k7gtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:04.452573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.452536 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:04.452573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.452573 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:04.452573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.452583 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7gtx\" (UniqueName: \"kubernetes.io/projected/db815bfe-85a0-48fb-858f-015cceef6ea6-kube-api-access-k7gtx\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:04.452768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:04.452592 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db815bfe-85a0-48fb-858f-015cceef6ea6-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:05.149233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:05.149176 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" Oct 30 14:27:05.149710 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:05.149238 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6" event={"ID":"db815bfe-85a0-48fb-858f-015cceef6ea6","Type":"ContainerDied","Data":"0114948bed20187994a77a4f34e43c3255a512bb1949c0c943df6751e696385f"} Oct 30 14:27:05.149710 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:05.149301 5824 scope.go:122] "RemoveContainer" containerID="e6e4a8820f531de05088f349cde66b8629319c4b91110e0bfe28e549e62c01fb" Oct 30 14:27:05.222427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:05.222401 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6"] Oct 30 14:27:05.235717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:05.235692 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-wn5g6"] Oct 30 14:27:06.318406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:06.318374 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" path="/var/lib/kubelet/pods/db815bfe-85a0-48fb-858f-015cceef6ea6/volumes" Oct 30 14:27:12.071583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:12.071500 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" podUID="d2df3c67-41d4-457c-8115-c6449e3d5fe8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:27:14.709693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:14.709669 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:27:14.799806 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:14.799773 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2df3c67-41d4-457c-8115-c6449e3d5fe8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df3c67-41d4-457c-8115-c6449e3d5fe8-plugins-config-volume\") pod \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\" (UID: \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\") " Oct 30 14:27:14.799988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:14.799932 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2df3c67-41d4-457c-8115-c6449e3d5fe8-kube-api-access-5wtx2\" (UniqueName: \"kubernetes.io/projected/d2df3c67-41d4-457c-8115-c6449e3d5fe8-kube-api-access-5wtx2\") pod \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\" (UID: \"d2df3c67-41d4-457c-8115-c6449e3d5fe8\") " Oct 30 14:27:14.800242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:14.800171 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2df3c67-41d4-457c-8115-c6449e3d5fe8-plugins-config-volume" pod "d2df3c67-41d4-457c-8115-c6449e3d5fe8" (UID: "d2df3c67-41d4-457c-8115-c6449e3d5fe8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:14.800726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:14.800700 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2df3c67-41d4-457c-8115-c6449e3d5fe8-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:14.802271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:14.802241 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2df3c67-41d4-457c-8115-c6449e3d5fe8-kube-api-access-5wtx2" pod "d2df3c67-41d4-457c-8115-c6449e3d5fe8" (UID: "d2df3c67-41d4-457c-8115-c6449e3d5fe8"). InnerVolumeSpecName "kube-api-access-5wtx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:14.902360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:14.902334 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wtx2\" (UniqueName: \"kubernetes.io/projected/d2df3c67-41d4-457c-8115-c6449e3d5fe8-kube-api-access-5wtx2\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:15.237074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.237041 5824 generic.go:356] "Generic (PLEG): container finished" podID="d2df3c67-41d4-457c-8115-c6449e3d5fe8" containerID="9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6" exitCode=0 Oct 30 14:27:15.237234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.237087 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" Oct 30 14:27:15.237234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.237123 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" event={"ID":"d2df3c67-41d4-457c-8115-c6449e3d5fe8","Type":"ContainerDied","Data":"9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6"} Oct 30 14:27:15.237234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.237177 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h" event={"ID":"d2df3c67-41d4-457c-8115-c6449e3d5fe8","Type":"ContainerDied","Data":"a6be0750a7b42ad8a9a352f89316a108e10d2e4f614c297c61fff4907792c5bb"} Oct 30 14:27:15.237234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.237220 5824 scope.go:122] "RemoveContainer" containerID="9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6" Oct 30 14:27:15.250398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.250372 5824 scope.go:122] "RemoveContainer" containerID="9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6" Oct 30 14:27:15.250765 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:27:15.250743 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6\": container with ID starting with 9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6 not found: ID does not exist" containerID="9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6" Oct 30 14:27:15.250825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.250771 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6"} err="failed to get container status \"9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6\": rpc error: code = NotFound desc = could not find container \"9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6\": container with ID starting with 9004dece5b5f9f971a9b8f2cdde01387a86b760226882854e20ee1d9658f62d6 not found: ID does not exist" Oct 30 14:27:15.314936 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.314904 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h"] Oct 30 14:27:15.328268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:15.328218 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca02c9-594d5d6fff-mvr9h"] Oct 30 14:27:16.317810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:16.317778 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2df3c67-41d4-457c-8115-c6449e3d5fe8" path="/var/lib/kubelet/pods/d2df3c67-41d4-457c-8115-c6449e3d5fe8/volumes" Oct 30 14:27:33.642072 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:27:33.642039 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:27:39.991493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:39.991459 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv"] Oct 30 14:27:39.991979 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:39.991728 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" containerID="cri-o://02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32" gracePeriod=30 Oct 30 14:27:45.280430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.280361 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:27:45.433315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.433273 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-data\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.433551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.433323 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03d32f51-9414-44e3-8ea0-556d90423045-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d32f51-9414-44e3-8ea0-556d90423045-istio-podinfo\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.433551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.433439 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-certs\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.433551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.433508 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-envoy\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.433551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.433514 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-data" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:45.433829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.433565 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-credential-socket\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.433885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.433818 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-certs" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:45.434005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.433980 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-credential-socket" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:45.434166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.434138 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-socket\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.434447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.434410 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-socket" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:45.434563 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.434534 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-kube-api-access-9s8q2\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-kube-api-access-9s8q2\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.434633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.434582 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-istio-token\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-istio-token\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.434751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.434734 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d32f51-9414-44e3-8ea0-556d90423045-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d32f51-9414-44e3-8ea0-556d90423045-istiod-ca-cert\") pod \"03d32f51-9414-44e3-8ea0-556d90423045\" (UID: \"03d32f51-9414-44e3-8ea0-556d90423045\") " Oct 30 14:27:45.435559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.435166 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d32f51-9414-44e3-8ea0-556d90423045-istiod-ca-cert" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:27:45.435715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.435689 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.435788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.435722 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.435788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.435740 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d32f51-9414-44e3-8ea0-556d90423045-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.435788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.435755 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.435788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.435769 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.435955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.435919 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-envoy" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:27:45.436914 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.436893 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-kube-api-access-9s8q2" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "kube-api-access-9s8q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:45.437046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.437024 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03d32f51-9414-44e3-8ea0-556d90423045-istio-podinfo" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:27:45.437425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.437406 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-istio-token" pod "03d32f51-9414-44e3-8ea0-556d90423045" (UID: "03d32f51-9414-44e3-8ea0-556d90423045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:27:45.537051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.536970 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d32f51-9414-44e3-8ea0-556d90423045-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.537051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.537001 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s8q2\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-kube-api-access-9s8q2\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.537051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.537011 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d32f51-9414-44e3-8ea0-556d90423045-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.537051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.537019 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d32f51-9414-44e3-8ea0-556d90423045-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:27:45.545046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.545015 5824 generic.go:356] "Generic (PLEG): container finished" podID="03d32f51-9414-44e3-8ea0-556d90423045" containerID="02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32" exitCode=0 Oct 30 14:27:45.545219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.545138 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" Oct 30 14:27:45.545219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.545156 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" event={"ID":"03d32f51-9414-44e3-8ea0-556d90423045","Type":"ContainerDied","Data":"02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32"} Oct 30 14:27:45.545335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.545266 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv" event={"ID":"03d32f51-9414-44e3-8ea0-556d90423045","Type":"ContainerDied","Data":"2ed20396dc77e4c5489eb7c7ec2c144cebb25cefef071fac30092f6c3c01eb4f"} Oct 30 14:27:45.545335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.545305 5824 scope.go:122] "RemoveContainer" containerID="02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32" Oct 30 14:27:45.560043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.560023 5824 scope.go:122] "RemoveContainer" containerID="f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c" Oct 30 14:27:45.572351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.572331 5824 scope.go:122] "RemoveContainer" containerID="02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32" Oct 30 14:27:45.572728 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:27:45.572707 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32\": container with ID starting with 02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32 not found: ID does not exist" containerID="02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32" Oct 30 14:27:45.572827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.572740 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32"} err="failed to get container status \"02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32\": rpc error: code = NotFound desc = could not find container \"02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32\": container with ID starting with 02fd23bd700fb5d11c3c03e96cdda45f98e03b9e59c18edd755cd0c1a2ad7d32 not found: ID does not exist" Oct 30 14:27:45.572827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.572765 5824 scope.go:122] "RemoveContainer" containerID="f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c" Oct 30 14:27:45.573087 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:27:45.573071 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c\": container with ID starting with f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c not found: ID does not exist" containerID="f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c" Oct 30 14:27:45.573145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.573096 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c"} err="failed to get container status \"f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c\": rpc error: code = NotFound desc = could not find container \"f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c\": container with ID starting with f5092fcb21724606622a89370aa5efbdb58c299660ba19eebff0e1fe456e684c not found: ID does not exist" Oct 30 14:27:45.619368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.619336 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv"] Oct 30 14:27:45.624492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:45.624468 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ca02c9-openshift-default-67db76697-d8mwv"] Oct 30 14:27:46.332226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:46.331167 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d32f51-9414-44e3-8ea0-556d90423045" path="/var/lib/kubelet/pods/03d32f51-9414-44e3-8ea0-556d90423045/volumes" Oct 30 14:27:48.578076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.578002 5824 generic.go:356] "Generic (PLEG): container finished" podID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerID="306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8" exitCode=0 Oct 30 14:27:48.578538 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.578078 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" event={"ID":"f05b8bdf-8389-4448-b6fa-2b995ff94f36","Type":"ContainerDied","Data":"306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8"} Oct 30 14:27:48.578636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.578617 5824 scope.go:122] "RemoveContainer" containerID="306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8" Oct 30 14:27:48.580595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.580562 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:48.580728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.580657 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:48.580728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.580713 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:27:48.591199 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.591169 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:27:48.928149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.928118 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk"] Oct 30 14:27:48.928940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.928914 5824 cpu_manager.go:414] "containerMap: removing container" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" Oct 30 14:27:48.928940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.928933 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.928954 5824 cpu_manager.go:414] "containerMap: removing container" podUID="d2df3c67-41d4-457c-8115-c6449e3d5fe8" containerName="payload-pre-processing" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.928960 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2df3c67-41d4-457c-8115-c6449e3d5fe8" containerName="payload-pre-processing" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.928980 5824 cpu_manager.go:414] "containerMap: removing container" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.928988 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.928996 5824 cpu_manager.go:414] "containerMap: removing container" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929000 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929008 5824 cpu_manager.go:414] "containerMap: removing container" podUID="65da2b41-a7a1-47b2-89ce-424fba8b82a5" containerName="payload-processing" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929013 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65da2b41-a7a1-47b2-89ce-424fba8b82a5" containerName="payload-processing" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929029 5824 cpu_manager.go:414] "containerMap: removing container" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929033 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929138 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2df3c67-41d4-457c-8115-c6449e3d5fe8" containerName="payload-pre-processing" Oct 30 14:27:48.929139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929148 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="db815bfe-85a0-48fb-858f-015cceef6ea6" containerName="istio-proxy" Oct 30 14:27:48.929807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929154 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" Oct 30 14:27:48.929807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929160 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="65da2b41-a7a1-47b2-89ce-424fba8b82a5" containerName="payload-processing" Oct 30 14:27:48.929807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.929167 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="629544b6-29fe-475f-8833-bca9cca08552" containerName="istio-proxy" Oct 30 14:27:48.932988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.932967 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:48.946902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.946880 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk"] Oct 30 14:27:48.951547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.951522 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-acefbb-dockercfg-zmmdj\"" Oct 30 14:27:48.951645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:48.951502 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-acefbb\"" Oct 30 14:27:49.000602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.000048 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk\" (UID: \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:49.000602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.000409 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4twfp\" (UniqueName: \"kubernetes.io/projected/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-kube-api-access-4twfp\") pod \"payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk\" (UID: \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:49.014910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.014873 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd"] Oct 30 14:27:49.016321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.016290 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d32f51-9414-44e3-8ea0-556d90423045" containerName="istio-proxy" Oct 30 14:27:49.027320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.020716 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:49.049335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.049303 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd"] Oct 30 14:27:49.107213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.107134 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk\" (UID: \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:49.120219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.109751 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b77a2a8-2324-44f6-a28e-97170eca645c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd\" (UID: \"8b77a2a8-2324-44f6-a28e-97170eca645c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:49.120219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.109992 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk\" (UID: \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:49.120219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.110040 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4twfp\" (UniqueName: \"kubernetes.io/projected/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-kube-api-access-4twfp\") pod \"payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk\" (UID: \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:49.120219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.110149 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdwj\" (UniqueName: \"kubernetes.io/projected/8b77a2a8-2324-44f6-a28e-97170eca645c-kube-api-access-scdwj\") pod \"payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd\" (UID: \"8b77a2a8-2324-44f6-a28e-97170eca645c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:49.164215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.154417 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4twfp\" (UniqueName: \"kubernetes.io/projected/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-kube-api-access-4twfp\") pod \"payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk\" (UID: \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:49.211573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.211495 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdwj\" (UniqueName: \"kubernetes.io/projected/8b77a2a8-2324-44f6-a28e-97170eca645c-kube-api-access-scdwj\") pod \"payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd\" (UID: \"8b77a2a8-2324-44f6-a28e-97170eca645c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:49.211730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.211593 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b77a2a8-2324-44f6-a28e-97170eca645c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd\" (UID: \"8b77a2a8-2324-44f6-a28e-97170eca645c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:49.212353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.212332 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b77a2a8-2324-44f6-a28e-97170eca645c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd\" (UID: \"8b77a2a8-2324-44f6-a28e-97170eca645c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:49.238283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.238253 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdwj\" (UniqueName: \"kubernetes.io/projected/8b77a2a8-2324-44f6-a28e-97170eca645c-kube-api-access-scdwj\") pod \"payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd\" (UID: \"8b77a2a8-2324-44f6-a28e-97170eca645c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:49.252484 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.252460 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:49.392351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.392308 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:49.579262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.577462 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk"] Oct 30 14:27:49.591990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.591957 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" event={"ID":"f05b8bdf-8389-4448-b6fa-2b995ff94f36","Type":"ContainerStarted","Data":"047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1"} Oct 30 14:27:49.708877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:49.708853 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd"] Oct 30 14:27:49.709594 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:27:49.709570 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b77a2a8_2324_44f6_a28e_97170eca645c.slice/crio-15b865fc9846eea860c13d198d1b72abd2859f5d3351b5c012b40eb6ecb8e91e WatchSource:0}: Error finding container 15b865fc9846eea860c13d198d1b72abd2859f5d3351b5c012b40eb6ecb8e91e: Status 404 returned error can't find the container with id 15b865fc9846eea860c13d198d1b72abd2859f5d3351b5c012b40eb6ecb8e91e Oct 30 14:27:50.267213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.266554 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:27:50.268381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.268339 5824 patch_prober.go:29] interesting pod/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 14:27:50.268535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.268397 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 14:27:50.601883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.601786 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" event={"ID":"8b77a2a8-2324-44f6-a28e-97170eca645c","Type":"ContainerStarted","Data":"ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4"} Oct 30 14:27:50.601883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.601852 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" event={"ID":"8b77a2a8-2324-44f6-a28e-97170eca645c","Type":"ContainerStarted","Data":"15b865fc9846eea860c13d198d1b72abd2859f5d3351b5c012b40eb6ecb8e91e"} Oct 30 14:27:50.601883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.601878 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:27:50.603550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.603524 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" event={"ID":"e4d62c9a-1dae-4303-8e00-cf0ce0a20268","Type":"ContainerStarted","Data":"2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027"} Oct 30 14:27:50.603686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.603577 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" event={"ID":"e4d62c9a-1dae-4303-8e00-cf0ce0a20268","Type":"ContainerStarted","Data":"97c170f9a85c4d3b96c4e797bd1d0a7ee7c2ae46e9c9a089f43d89cfd956006e"} Oct 30 14:27:50.603686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.603608 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:27:50.712233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.712170 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" podStartSLOduration=2.712158031 podStartE2EDuration="2.712158031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:50.667439585 +0000 UTC m=+2300.467945031" watchObservedRunningTime="2026-10-30 14:27:50.712158031 +0000 UTC m=+2300.512663497" Oct 30 14:27:50.712947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:50.712916 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" podStartSLOduration=2.712907215 podStartE2EDuration="2.712907215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:27:50.711390691 +0000 UTC m=+2300.511896137" watchObservedRunningTime="2026-10-30 14:27:50.712907215 +0000 UTC m=+2300.513412659" Oct 30 14:27:51.260575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:51.260539 5824 patch_prober.go:29] interesting pod/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 14:27:51.260883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:51.260850 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 14:27:52.261579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:52.261542 5824 patch_prober.go:29] interesting pod/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 14:27:52.262171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:52.262141 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 14:27:53.260628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:53.260593 5824 patch_prober.go:29] interesting pod/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 14:27:53.260923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:53.260894 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 14:27:54.266446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:54.266405 5824 patch_prober.go:29] interesting pod/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 14:27:54.267228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:54.267172 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 14:27:55.265347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:55.265315 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:27:55.265863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:55.265834 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:27:55.266755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:27:55.266737 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:28:00.387795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.387759 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t"] Oct 30 14:28:00.392777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.392753 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.425735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.425701 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f3c580-openshift-default-dockercfg-ws8ll\"" Oct 30 14:28:00.438227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.438163 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t"] Oct 30 14:28:00.505449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.505414 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd369177-a300-45ac-a896-65e097ce54ed-istiod-ca-cert\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.505629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.505486 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-credential-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.505629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.505609 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bp7\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-kube-api-access-f7bp7\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.505781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.505746 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-envoy\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.505919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.505818 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-certs\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.505974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.505950 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-data\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.506024 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.506009 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-istio-token\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.506077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.506067 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd369177-a300-45ac-a896-65e097ce54ed-istio-podinfo\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.506229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.506176 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.607581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.607544 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-envoy\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.607784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.607591 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-certs\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.607784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.607699 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-data\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.607784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.607754 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-istio-token\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.607968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.607811 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd369177-a300-45ac-a896-65e097ce54ed-istio-podinfo\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.607968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.607872 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.607968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.607915 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd369177-a300-45ac-a896-65e097ce54ed-istiod-ca-cert\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.608212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.607982 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-credential-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.608212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.608044 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bp7\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-kube-api-access-f7bp7\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.608212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.608092 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-data\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.608212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.608042 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-certs\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.608413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.608308 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-credential-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.608413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.608356 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-socket\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.608649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.608630 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd369177-a300-45ac-a896-65e097ce54ed-istiod-ca-cert\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.609946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.609913 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-envoy\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.610271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.610248 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd369177-a300-45ac-a896-65e097ce54ed-istio-podinfo\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.648355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.648239 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-istio-token\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.649163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.649133 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bp7\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-kube-api-access-f7bp7\") pod \"e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:00.728071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:00.728038 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:01.054370 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:28:01.054337 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd369177_a300_45ac_a896_65e097ce54ed.slice/crio-113472b30108226a2a13eb554992a30c0767db56f451118afd246ab4ef79cfdb WatchSource:0}: Error finding container 113472b30108226a2a13eb554992a30c0767db56f451118afd246ab4ef79cfdb: Status 404 returned error can't find the container with id 113472b30108226a2a13eb554992a30c0767db56f451118afd246ab4ef79cfdb Oct 30 14:28:01.056341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.056317 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t"] Oct 30 14:28:01.057319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.057286 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:01.057385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.057355 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:01.057425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.057384 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:01.614524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.614498 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:28:01.615007 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.614846 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:28:01.694606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.694571 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" event={"ID":"cd369177-a300-45ac-a896-65e097ce54ed","Type":"ContainerStarted","Data":"5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db"} Oct 30 14:28:01.694793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.694632 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" event={"ID":"cd369177-a300-45ac-a896-65e097ce54ed","Type":"ContainerStarted","Data":"113472b30108226a2a13eb554992a30c0767db56f451118afd246ab4ef79cfdb"} Oct 30 14:28:01.729055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:01.729022 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:02.284698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.284634 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" podStartSLOduration=2.284620481 podStartE2EDuration="2.284620481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:01.77257731 +0000 UTC m=+2311.573082755" watchObservedRunningTime="2026-10-30 14:28:02.284620481 +0000 UTC m=+2312.085125925" Oct 30 14:28:02.285450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.285425 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm"] Oct 30 14:28:02.290379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.290361 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.311267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.311242 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-93549e-openshift-default-dockercfg-r6j9x\"" Oct 30 14:28:02.324104 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.324055 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm"] Oct 30 14:28:02.437103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437063 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbj94\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-kube-api-access-jbj94\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.437313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437119 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e05408-2277-4bfd-a618-b8cbd2249911-istiod-ca-cert\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.437364 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437327 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-envoy\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.437402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437363 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-credential-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.437509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437496 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-data\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.437550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437521 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.437667 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437656 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-certs\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.437704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437675 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-token\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.437789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.437771 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-podinfo\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539239 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-envoy\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539291 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-credential-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539423 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-data\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539456 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539597 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-certs\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539627 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-token\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539691 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-podinfo\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539742 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbj94\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-kube-api-access-jbj94\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.539857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.539792 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e05408-2277-4bfd-a618-b8cbd2249911-istiod-ca-cert\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.540986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.540137 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.540986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.540612 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e05408-2277-4bfd-a618-b8cbd2249911-istiod-ca-cert\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.540986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.540843 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-certs\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.575213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.541348 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-data\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.575213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.559525 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-credential-socket\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.575213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.559922 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-podinfo\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.578141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.578108 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-envoy\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.580657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.580629 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbj94\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-kube-api-access-jbj94\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.580959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.580935 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-token\") pod \"e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.605440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.605416 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:02.730266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.730075 5824 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:28:02.730266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.730155 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 14:28:02.971406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.971367 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm"] Oct 30 14:28:02.971832 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:28:02.971804 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e05408_2277_4bfd_a618_b8cbd2249911.slice/crio-29a56e3635afbbb122c07f5b527770eb908f5650bc9c8599c575d0b6091e6a9b WatchSource:0}: Error finding container 29a56e3635afbbb122c07f5b527770eb908f5650bc9c8599c575d0b6091e6a9b: Status 404 returned error can't find the container with id 29a56e3635afbbb122c07f5b527770eb908f5650bc9c8599c575d0b6091e6a9b Oct 30 14:28:02.974572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.974539 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:02.974677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.974604 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:02.974677 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:02.974632 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:03.672210 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:28:03.672147 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:28:03.714338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:03.714289 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" event={"ID":"a0e05408-2277-4bfd-a618-b8cbd2249911","Type":"ContainerStarted","Data":"0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6"} Oct 30 14:28:03.714535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:03.714355 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" event={"ID":"a0e05408-2277-4bfd-a618-b8cbd2249911","Type":"ContainerStarted","Data":"29a56e3635afbbb122c07f5b527770eb908f5650bc9c8599c575d0b6091e6a9b"} Oct 30 14:28:03.728804 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:03.728776 5824 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:28:03.728910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:03.728822 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 14:28:03.789214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:03.788837 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podStartSLOduration=1.7888160690000001 podStartE2EDuration="1.788816069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:03.763428918 +0000 UTC m=+2313.563934364" watchObservedRunningTime="2026-10-30 14:28:03.788816069 +0000 UTC m=+2313.589321514" Oct 30 14:28:04.606416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:04.606374 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:04.607637 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:04.607615 5824 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 14:28:04.607719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:04.607656 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 14:28:04.729214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:04.728522 5824 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:28:04.729214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:04.728590 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 14:28:05.606286 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:05.606248 5824 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 14:28:05.606682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:05.606304 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 14:28:05.729152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:05.729114 5824 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:28:05.729314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:05.729166 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 14:28:05.995218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:05.994012 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj"] Oct 30 14:28:06.004233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.003322 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj"] Oct 30 14:28:06.004233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.003451 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.022432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.022407 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d15602-openshift-default-dockercfg-bws4s\"" Oct 30 14:28:06.130833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.130741 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-credential-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.130833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.130783 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-envoy\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.131039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.131019 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqrj\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-kube-api-access-tmqrj\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.131124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.131094 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-data\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.131164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.131137 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-podinfo\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.131217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.131170 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-certs\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.131250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.131215 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istiod-ca-cert\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.131291 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.131257 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-token\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.131325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.131299 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.232860 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqrj\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-kube-api-access-tmqrj\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.232941 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-data\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.232965 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-podinfo\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.232991 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-certs\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.233016 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istiod-ca-cert\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.233053 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-token\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.233096 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.233280 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-credential-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.233791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.233325 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-envoy\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.234489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.234411 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-data\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.234857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.234777 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-certs\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.234857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.234835 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.235442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.235177 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istiod-ca-cert\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.235442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.235221 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-credential-socket\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.237479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.237459 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-envoy\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.238045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.238025 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-podinfo\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.311633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.311528 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqrj\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-kube-api-access-tmqrj\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.313494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.313464 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-token\") pod \"e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.325842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.325797 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:06.712803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.712280 5824 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 14:28:06.712803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.712342 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 14:28:06.732135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.731767 5824 patch_prober.go:29] interesting pod/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 14:28:06.732135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.731832 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 14:28:06.899598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.899558 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:06.899701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.899645 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:06.899701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.899678 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:06.901085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:06.901059 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj"] Oct 30 14:28:07.606167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:07.606134 5824 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 14:28:07.606402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:07.606221 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 14:28:07.824166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:07.824111 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" event={"ID":"da87cb61-ccfa-49c2-9c39-cd6583a00d3c","Type":"ContainerStarted","Data":"0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207"} Oct 30 14:28:07.824608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:07.824207 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" event={"ID":"da87cb61-ccfa-49c2-9c39-cd6583a00d3c","Type":"ContainerStarted","Data":"036bf5cff7a1e7930cf25a4bb1fd67f9bb7c26e2804be8369efdb169d79fa05f"} Oct 30 14:28:07.871292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:07.870461 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:07.872327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:07.872291 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:07.874539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:07.874513 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:28:07.899549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:07.899505 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podStartSLOduration=2.8994923679999998 podStartE2EDuration="2.899492368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:07.896107904 +0000 UTC m=+2317.696613353" watchObservedRunningTime="2026-10-30 14:28:07.899492368 +0000 UTC m=+2317.699997809" Oct 30 14:28:08.326432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:08.326406 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:08.328109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:08.328077 5824 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 14:28:08.328257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:08.328164 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 14:28:08.606235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:08.606138 5824 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 14:28:08.606235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:08.606212 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 14:28:09.327373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:09.327339 5824 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 14:28:09.327782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:09.327394 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 14:28:09.605765 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:09.605694 5824 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 14:28:09.605765 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:09.605737 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 14:28:10.327316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:10.327288 5824 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 14:28:10.327491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:10.327339 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 14:28:10.610925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:10.610846 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:10.611295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:10.611272 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:10.612082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:10.612061 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:28:11.327381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:11.327344 5824 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 14:28:11.327559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:11.327407 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 14:28:12.327200 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:12.327163 5824 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 14:28:12.327377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:12.327251 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 14:28:13.326732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:13.326692 5824 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 14:28:13.327250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:13.326762 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 14:28:14.337411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:14.337377 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:14.338680 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:14.338646 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:14.339464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:14.339443 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:28:19.921746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:19.921709 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq"] Oct 30 14:28:19.927846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:19.927823 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:19.950137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:19.950113 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1d3696-dockercfg-crvv4\"" Oct 30 14:28:19.964431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:19.964405 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1d3696\"" Oct 30 14:28:19.970233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:19.970211 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq"] Oct 30 14:28:20.233002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:20.232964 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zw9b\" (UniqueName: \"kubernetes.io/projected/98701cd7-fc28-4901-8076-1bd0e8c6f26c-kube-api-access-2zw9b\") pod \"payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq\" (UID: \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:20.233243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:20.233104 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98701cd7-fc28-4901-8076-1bd0e8c6f26c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq\" (UID: \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:20.334602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:20.334568 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zw9b\" (UniqueName: \"kubernetes.io/projected/98701cd7-fc28-4901-8076-1bd0e8c6f26c-kube-api-access-2zw9b\") pod \"payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq\" (UID: \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:20.334751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:20.334655 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98701cd7-fc28-4901-8076-1bd0e8c6f26c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq\" (UID: \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:20.335385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:20.335364 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98701cd7-fc28-4901-8076-1bd0e8c6f26c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq\" (UID: \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:20.360631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:20.360605 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zw9b\" (UniqueName: \"kubernetes.io/projected/98701cd7-fc28-4901-8076-1bd0e8c6f26c-kube-api-access-2zw9b\") pod \"payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq\" (UID: \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:20.545627 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:20.545561 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:20.883535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:20.883499 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq"] Oct 30 14:28:20.883729 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:28:20.883657 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98701cd7_fc28_4901_8076_1bd0e8c6f26c.slice/crio-d02410411baf6fadf99e19f7c2726013a6dbb46d3e90c16faff1b0b0b8f440a4 WatchSource:0}: Error finding container d02410411baf6fadf99e19f7c2726013a6dbb46d3e90c16faff1b0b0b8f440a4: Status 404 returned error can't find the container with id d02410411baf6fadf99e19f7c2726013a6dbb46d3e90c16faff1b0b0b8f440a4 Oct 30 14:28:21.170982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:21.170888 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" event={"ID":"98701cd7-fc28-4901-8076-1bd0e8c6f26c","Type":"ContainerStarted","Data":"310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f"} Oct 30 14:28:21.170982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:21.170954 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" event={"ID":"98701cd7-fc28-4901-8076-1bd0e8c6f26c","Type":"ContainerStarted","Data":"d02410411baf6fadf99e19f7c2726013a6dbb46d3e90c16faff1b0b0b8f440a4"} Oct 30 14:28:21.170982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:21.170982 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:21.218521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:21.218456 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" podStartSLOduration=2.2184365169999998 podStartE2EDuration="2.218436517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:21.216207633 +0000 UTC m=+2331.016713274" watchObservedRunningTime="2026-10-30 14:28:21.218436517 +0000 UTC m=+2331.018941962" Oct 30 14:28:21.982051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:21.982018 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq"] Oct 30 14:28:21.982275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:21.982247 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" containerID="cri-o://81a3fd96841822e96f44771cc22a0d01b8c801efe057f9c0359c2bf1fb79278a" gracePeriod=30 Oct 30 14:28:27.332292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.330895 5824 generic.go:356] "Generic (PLEG): container finished" podID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerID="81a3fd96841822e96f44771cc22a0d01b8c801efe057f9c0359c2bf1fb79278a" exitCode=0 Oct 30 14:28:27.332292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.330974 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" event={"ID":"a307e4c8-46bf-44bb-ba8c-1dbb15871f61","Type":"ContainerDied","Data":"81a3fd96841822e96f44771cc22a0d01b8c801efe057f9c0359c2bf1fb79278a"} Oct 30 14:28:27.390599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.390569 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:28:27.499058 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499011 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-data\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499138 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-kube-api-access-p65jz\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-kube-api-access-p65jz\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499211 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-envoy\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499239 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-credential-socket\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499266 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-token\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-token\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499365 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-certs\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499418 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-podinfo\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499499 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istiod-ca-cert\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499560 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-socket\") pod \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\" (UID: \"a307e4c8-46bf-44bb-ba8c-1dbb15871f61\") " Oct 30 14:28:27.499731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499654 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-credential-socket" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:27.499731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499712 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-data" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:27.499968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499944 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-certs" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:27.500113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.499960 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istiod-ca-cert" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:27.500319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.500297 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-socket" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:27.500421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.500385 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:27.500421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.500403 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:27.500421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.500412 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:27.500758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.500427 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:27.500758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.500436 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:27.507684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.507641 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-podinfo" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:27.507684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.507652 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-kube-api-access-p65jz" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "kube-api-access-p65jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:27.507684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.507648 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-envoy" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:27.508931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.508909 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-token" pod "a307e4c8-46bf-44bb-ba8c-1dbb15871f61" (UID: "a307e4c8-46bf-44bb-ba8c-1dbb15871f61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:27.602396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.602321 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p65jz\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-kube-api-access-p65jz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:27.602396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.602355 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:27.602396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.602370 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:27.602396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:27.602383 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307e4c8-46bf-44bb-ba8c-1dbb15871f61-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:28.339686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:28.339656 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" Oct 30 14:28:28.340048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:28.339680 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq" event={"ID":"a307e4c8-46bf-44bb-ba8c-1dbb15871f61","Type":"ContainerDied","Data":"583075c8642a794aceb03f1a569f74c6e1794e03341cde66040e7c381c865ca7"} Oct 30 14:28:28.340048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:28.339739 5824 scope.go:122] "RemoveContainer" containerID="81a3fd96841822e96f44771cc22a0d01b8c801efe057f9c0359c2bf1fb79278a" Oct 30 14:28:28.418360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:28.418332 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq"] Oct 30 14:28:28.431908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:28.431883 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jvvbq"] Oct 30 14:28:28.551852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:28.551818 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn"] Oct 30 14:28:28.552015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:28.551996 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" containerID="cri-o://a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b" gracePeriod=30 Oct 30 14:28:30.319008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:30.318974 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" path="/var/lib/kubelet/pods/a307e4c8-46bf-44bb-ba8c-1dbb15871f61/volumes" Oct 30 14:28:32.188492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:32.188464 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:28:32.380481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:32.380443 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:28:32.380647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:32.380509 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:28:33.645637 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:28:33.645603 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:28:33.859075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.859048 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:28:33.893755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.893730 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-token\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-token\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.893883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.893781 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-certs\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.893883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.893817 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-envoy\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.893883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.893868 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istiod-ca-cert\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.894030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.893888 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-data\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.894030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.893971 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-socket\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.894030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.894012 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-podinfo\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.894699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.894041 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-kube-api-access-knjrk\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-kube-api-access-knjrk\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.894699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.894279 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-certs" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.894699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.894296 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-credential-socket\") pod \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\" (UID: \"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9\") " Oct 30 14:28:33.894699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.894254 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-data" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.894699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.894328 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istiod-ca-cert" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:28:33.894699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.894300 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-socket" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.894699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.894506 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-credential-socket" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.895151 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.895128 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.895245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.895157 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.895245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.895172 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.895245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.895200 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.895245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.895214 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.896122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.896099 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-token" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:33.896220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.896127 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-envoy" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:28:33.896700 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.896681 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-podinfo" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:28:33.896798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.896700 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-kube-api-access-knjrk" pod "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" (UID: "68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9"). InnerVolumeSpecName "kube-api-access-knjrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:28:33.996565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.996526 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.996565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.996562 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.996565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.996571 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:33.996565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:33.996580 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knjrk\" (UniqueName: \"kubernetes.io/projected/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9-kube-api-access-knjrk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:28:34.385724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.385692 5824 generic.go:356] "Generic (PLEG): container finished" podID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerID="a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b" exitCode=0 Oct 30 14:28:34.385870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.385738 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" Oct 30 14:28:34.385870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.385759 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" event={"ID":"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9","Type":"ContainerDied","Data":"a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b"} Oct 30 14:28:34.385870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.385796 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn" event={"ID":"68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9","Type":"ContainerDied","Data":"258d07766d872dbc826f6fc221c0e9dd2114810c63edce002ea8f0fe1cca0e43"} Oct 30 14:28:34.385870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.385815 5824 scope.go:122] "RemoveContainer" containerID="a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b" Oct 30 14:28:34.396717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.396696 5824 scope.go:122] "RemoveContainer" containerID="a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b" Oct 30 14:28:34.396992 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:28:34.396972 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b\": container with ID starting with a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b not found: ID does not exist" containerID="a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b" Oct 30 14:28:34.397041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.396999 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b"} err="failed to get container status \"a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b\": rpc error: code = NotFound desc = could not find container \"a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b\": container with ID starting with a919d67fd2d58cf679efd4b0f27f5548ff45aca53009496827a1e5cd3e498f3b not found: ID does not exist" Oct 30 14:28:34.463628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.463604 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn"] Oct 30 14:28:34.476940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:34.476917 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-x52tn"] Oct 30 14:28:35.558502 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.558470 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm"] Oct 30 14:28:35.559316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.559299 5824 cpu_manager.go:414] "containerMap: removing container" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" Oct 30 14:28:35.559373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.559318 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" Oct 30 14:28:35.559373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.559343 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" Oct 30 14:28:35.559373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.559349 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" Oct 30 14:28:35.559471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.559457 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" containerName="istio-proxy" Oct 30 14:28:35.559471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.559468 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a307e4c8-46bf-44bb-ba8c-1dbb15871f61" containerName="istio-proxy" Oct 30 14:28:35.564817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.564798 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.584686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.584663 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm"] Oct 30 14:28:35.593077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.593052 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-acefbb-serving-cert\"" Oct 30 14:28:35.616674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.616646 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.616869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.616846 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m4mv\" (UniqueName: \"kubernetes.io/projected/fb511c01-90a9-465a-8b03-830417169b98-kube-api-access-8m4mv\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.616950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.616900 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.617085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.617057 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb511c01-90a9-465a-8b03-830417169b98-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.718544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.718517 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m4mv\" (UniqueName: \"kubernetes.io/projected/fb511c01-90a9-465a-8b03-830417169b98-kube-api-access-8m4mv\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.718702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.718561 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.718702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.718623 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb511c01-90a9-465a-8b03-830417169b98-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.718702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.718682 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.719460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.719434 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb511c01-90a9-465a-8b03-830417169b98-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.721074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.721051 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.721171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.721143 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.744609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.744582 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m4mv\" (UniqueName: \"kubernetes.io/projected/fb511c01-90a9-465a-8b03-830417169b98-kube-api-access-8m4mv\") pod \"maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:35.880025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:35.879930 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:36.192958 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:28:36.192931 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb511c01_90a9_465a_8b03_830417169b98.slice/crio-6aee3028708dcff412763d6f9320bff9955ce0a929bed9efa7be7e6d567f2894 WatchSource:0}: Error finding container 6aee3028708dcff412763d6f9320bff9955ce0a929bed9efa7be7e6d567f2894: Status 404 returned error can't find the container with id 6aee3028708dcff412763d6f9320bff9955ce0a929bed9efa7be7e6d567f2894 Oct 30 14:28:36.194088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:36.194067 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm"] Oct 30 14:28:36.194424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:36.194406 5824 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:28:36.318385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:36.318357 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9" path="/var/lib/kubelet/pods/68ece2f2-3dfd-4b6f-b36b-b1f8cbfd97e9/volumes" Oct 30 14:28:36.402475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:36.402447 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" event={"ID":"fb511c01-90a9-465a-8b03-830417169b98","Type":"ContainerStarted","Data":"6aee3028708dcff412763d6f9320bff9955ce0a929bed9efa7be7e6d567f2894"} Oct 30 14:28:37.137119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.137084 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt"] Oct 30 14:28:37.141975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.141955 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.172367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.172339 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0b1c15-openshift-default-dockercfg-bmcps\"" Oct 30 14:28:37.176797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.176772 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt"] Oct 30 14:28:37.241962 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.241933 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-data\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.242101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.242073 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnp29\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-kube-api-access-dnp29\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.242171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.242117 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-certs\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.242267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.242249 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-token\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.242326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.242298 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-credential-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.242375 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.242335 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-envoy\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.242511 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.242486 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-podinfo\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.242562 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.242535 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istiod-ca-cert\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.242648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.242633 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.343594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.343568 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-envoy\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.343712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.343673 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-podinfo\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.343778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.343715 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istiod-ca-cert\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.343778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.343768 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.343892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.343800 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-data\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.343892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.343876 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnp29\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-kube-api-access-dnp29\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.344005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.343914 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-certs\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.344005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.343984 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-token\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.344117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.344059 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-credential-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.344169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.344127 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.344503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.344476 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-credential-socket\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.344596 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.344504 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-data\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.344596 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.344558 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-certs\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.344715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.344590 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istiod-ca-cert\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.345980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.345957 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-envoy\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.346278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.346261 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-podinfo\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.369961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.369933 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-token\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.370265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.370247 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnp29\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-kube-api-access-dnp29\") pod \"e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.412144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.412048 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" event={"ID":"fb511c01-90a9-465a-8b03-830417169b98","Type":"ContainerStarted","Data":"517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff"} Oct 30 14:28:37.412345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.412148 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:37.457651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.457618 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:37.461338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.461298 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" podStartSLOduration=2.107991336 podStartE2EDuration="2.461286153s" totalImagesPullingTime="353.294817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:37.458006937 +0000 UTC m=+2347.258512382" watchObservedRunningTime="2026-10-30 14:28:37.461286153 +0000 UTC m=+2347.261791644" Oct 30 14:28:37.797404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.796930 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt"] Oct 30 14:28:37.800761 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:28:37.800731 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587ac0b1_9b6f_47ee_9c1e_f924a0a62ffc.slice/crio-fe7e4a29bb815051175db257bc6b017c8b90bc8f48195a67bb3c418ea1fe611e WatchSource:0}: Error finding container fe7e4a29bb815051175db257bc6b017c8b90bc8f48195a67bb3c418ea1fe611e: Status 404 returned error can't find the container with id fe7e4a29bb815051175db257bc6b017c8b90bc8f48195a67bb3c418ea1fe611e Oct 30 14:28:37.804726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.804689 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:37.804820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.804779 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:37.804891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:37.804830 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:28:38.420646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:38.420614 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" event={"ID":"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc","Type":"ContainerStarted","Data":"1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897"} Oct 30 14:28:38.421140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:38.420679 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" event={"ID":"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc","Type":"ContainerStarted","Data":"fe7e4a29bb815051175db257bc6b017c8b90bc8f48195a67bb3c418ea1fe611e"} Oct 30 14:28:38.458265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:38.458234 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:38.459797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:38.459771 5824 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 14:28:38.459928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:38.459824 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 14:28:38.470974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:38.470928 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" podStartSLOduration=1.470915025 podStartE2EDuration="1.470915025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:28:38.469239963 +0000 UTC m=+2348.269745408" watchObservedRunningTime="2026-10-30 14:28:38.470915025 +0000 UTC m=+2348.271420469" Oct 30 14:28:39.459706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:39.459662 5824 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 14:28:39.460378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:39.460341 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 14:28:40.466553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:40.466516 5824 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 14:28:40.467057 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:40.466572 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 14:28:41.466271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:41.465298 5824 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 14:28:41.466271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:41.465364 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 14:28:42.461214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:42.460511 5824 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 14:28:42.461214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:42.460568 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 14:28:43.426542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:43.426511 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:28:43.461872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:43.461490 5824 patch_prober.go:29] interesting pod/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 14:28:43.461872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:43.461549 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 14:28:44.608105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:44.608075 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:44.608840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:44.608820 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:28:44.609992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:28:44.609966 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:29:03.643967 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:29:03.643929 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:29:33.655407 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:29:33.655370 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:29:39.520843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.520785 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6"] Oct 30 14:29:39.528729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.528702 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:39.578248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.566804 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-1d6347\"" Oct 30 14:29:39.578248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.566938 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-1d6347-dockercfg-dfgwf\"" Oct 30 14:29:39.578248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.574726 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6"] Oct 30 14:29:39.627673 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.615922 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0597f7-23b7-4be4-af5e-4082e092826b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6\" (UID: \"6f0597f7-23b7-4be4-af5e-4082e092826b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:39.632476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.632442 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssj54\" (UniqueName: \"kubernetes.io/projected/6f0597f7-23b7-4be4-af5e-4082e092826b-kube-api-access-ssj54\") pod \"payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6\" (UID: \"6f0597f7-23b7-4be4-af5e-4082e092826b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:39.736948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.736910 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssj54\" (UniqueName: \"kubernetes.io/projected/6f0597f7-23b7-4be4-af5e-4082e092826b-kube-api-access-ssj54\") pod \"payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6\" (UID: \"6f0597f7-23b7-4be4-af5e-4082e092826b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:39.737146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.737118 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0597f7-23b7-4be4-af5e-4082e092826b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6\" (UID: \"6f0597f7-23b7-4be4-af5e-4082e092826b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:39.737827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.737806 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0597f7-23b7-4be4-af5e-4082e092826b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6\" (UID: \"6f0597f7-23b7-4be4-af5e-4082e092826b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:39.766623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.766591 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssj54\" (UniqueName: \"kubernetes.io/projected/6f0597f7-23b7-4be4-af5e-4082e092826b-kube-api-access-ssj54\") pod \"payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6\" (UID: \"6f0597f7-23b7-4be4-af5e-4082e092826b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:39.915618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:39.915554 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:40.265844 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:29:40.265807 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0597f7_23b7_4be4_af5e_4082e092826b.slice/crio-4642363786edb09e35cdcd74b4ff07cd67a0f4287e03a6e7c19cfe0e85f59a10 WatchSource:0}: Error finding container 4642363786edb09e35cdcd74b4ff07cd67a0f4287e03a6e7c19cfe0e85f59a10: Status 404 returned error can't find the container with id 4642363786edb09e35cdcd74b4ff07cd67a0f4287e03a6e7c19cfe0e85f59a10 Oct 30 14:29:40.267002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:40.266967 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6"] Oct 30 14:29:41.029885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:41.029847 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" event={"ID":"6f0597f7-23b7-4be4-af5e-4082e092826b","Type":"ContainerStarted","Data":"4a895b9d800bed2155bd0abbb496f38a39f2d30bbaa2bce9979a95f77803c120"} Oct 30 14:29:41.030342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:41.029901 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" event={"ID":"6f0597f7-23b7-4be4-af5e-4082e092826b","Type":"ContainerStarted","Data":"4642363786edb09e35cdcd74b4ff07cd67a0f4287e03a6e7c19cfe0e85f59a10"} Oct 30 14:29:41.030342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:41.029921 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:41.076524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:41.076474 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" podStartSLOduration=2.076461549 podStartE2EDuration="2.076461549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:29:41.07395514 +0000 UTC m=+2410.874460586" watchObservedRunningTime="2026-10-30 14:29:41.076461549 +0000 UTC m=+2410.876966993" Oct 30 14:29:52.041049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:52.041020 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:29:52.298468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:52.298384 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg"] Oct 30 14:29:52.298646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:52.298622 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" containerID="cri-o://674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28" gracePeriod=30 Oct 30 14:29:56.184609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:56.184574 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hszvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:29:56.185015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:56.184640 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:29:57.589236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.589213 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:29:57.625253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625221 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-credential-socket\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.625384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625310 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-socket\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.625455 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625433 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-credential-socket" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:57.625513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625491 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-data\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.625570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625543 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09180ae1-ace5-43a7-9b01-635b157cdf4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09180ae1-ace5-43a7-9b01-635b157cdf4c-istiod-ca-cert\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.625629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625581 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-socket" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:57.625629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625585 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-certs\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.625748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625728 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-data" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:57.625807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625744 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-envoy\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.625807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625781 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-token\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-token\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.625893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625813 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-certs" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:57.625893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625839 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-podinfo\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.625893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625852 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09180ae1-ace5-43a7-9b01-635b157cdf4c-istiod-ca-cert" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:29:57.625893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.625887 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-kube-api-access-5sjnw\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-kube-api-access-5sjnw\") pod \"09180ae1-ace5-43a7-9b01-635b157cdf4c\" (UID: \"09180ae1-ace5-43a7-9b01-635b157cdf4c\") " Oct 30 14:29:57.626576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.626555 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09180ae1-ace5-43a7-9b01-635b157cdf4c-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:57.626701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.626579 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:57.626701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.626589 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:57.626701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.626598 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:57.626701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.626607 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:57.628033 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.628010 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-token" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:57.628126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.628029 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-podinfo" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:29:57.628273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.628248 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-envoy" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:29:57.628384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.628341 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-kube-api-access-5sjnw" pod "09180ae1-ace5-43a7-9b01-635b157cdf4c" (UID: "09180ae1-ace5-43a7-9b01-635b157cdf4c"). InnerVolumeSpecName "kube-api-access-5sjnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:29:57.727912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.727880 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:57.727912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.727908 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sjnw\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-kube-api-access-5sjnw\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:57.727912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.727919 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:57.728132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:57.727928 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09180ae1-ace5-43a7-9b01-635b157cdf4c-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:29:58.258066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.258031 5824 generic.go:356] "Generic (PLEG): container finished" podID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerID="674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28" exitCode=0 Oct 30 14:29:58.258272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.258083 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" Oct 30 14:29:58.258272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.258115 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" event={"ID":"09180ae1-ace5-43a7-9b01-635b157cdf4c","Type":"ContainerDied","Data":"674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28"} Oct 30 14:29:58.258272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.258171 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg" event={"ID":"09180ae1-ace5-43a7-9b01-635b157cdf4c","Type":"ContainerDied","Data":"9eb1699be5e046645bd65445586f6e01af3e71893878e244779c8c60d4f5f48c"} Oct 30 14:29:58.258272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.258223 5824 scope.go:122] "RemoveContainer" containerID="674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28" Oct 30 14:29:58.269774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.269750 5824 scope.go:122] "RemoveContainer" containerID="674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28" Oct 30 14:29:58.270074 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:29:58.270052 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28\": container with ID starting with 674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28 not found: ID does not exist" containerID="674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28" Oct 30 14:29:58.270158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.270081 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28"} err="failed to get container status \"674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28\": rpc error: code = NotFound desc = could not find container \"674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28\": container with ID starting with 674dd01b7b1609673674485fa3c393f5d8350736cc0811549c96f45a811ada28 not found: ID does not exist" Oct 30 14:29:58.331706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.331684 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg"] Oct 30 14:29:58.344945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:29:58.344925 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hszvg"] Oct 30 14:30:00.171613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.171576 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq"] Oct 30 14:30:00.173534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.173503 5824 cpu_manager.go:414] "containerMap: removing container" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" Oct 30 14:30:00.173534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.173527 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" Oct 30 14:30:00.173884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.173864 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" containerName="istio-proxy" Oct 30 14:30:00.179042 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.179021 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd"] Oct 30 14:30:00.179201 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.179144 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:00.184895 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.184868 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:00.190591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.190564 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd"] Oct 30 14:30:00.196243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.196163 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq"] Oct 30 14:30:00.201625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.201604 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl"] Oct 30 14:30:00.207080 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.207061 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:00.207461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.207444 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd"] Oct 30 14:30:00.212441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.212419 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:00.213463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.213428 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl"] Oct 30 14:30:00.235249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.235173 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:00.237246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.237182 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl"] Oct 30 14:30:00.243200 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.241666 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd"] Oct 30 14:30:00.250350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.246135 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl"] Oct 30 14:30:00.252264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.252244 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz"] Oct 30 14:30:00.260876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.260838 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlvm\" (UniqueName: \"kubernetes.io/projected/9a085f71-14d2-4210-b228-abd9eb70f463-kube-api-access-xqlvm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd\" (UID: \"9a085f71-14d2-4210-b228-abd9eb70f463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:00.261048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.261030 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl\" (UID: \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:00.261136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.261087 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd\" (UID: \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:00.261136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.261120 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frk2v\" (UniqueName: \"kubernetes.io/projected/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-kube-api-access-frk2v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl\" (UID: \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:00.261295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.261227 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxc7f\" (UniqueName: \"kubernetes.io/projected/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-kube-api-access-mxc7f\") pod \"maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd\" (UID: \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:00.261588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.261374 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca50db98-1579-42c1-8e66-1ebb6e7c5323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq\" (UID: \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:00.261588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.261440 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a085f71-14d2-4210-b228-abd9eb70f463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd\" (UID: \"9a085f71-14d2-4210-b228-abd9eb70f463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:00.261588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.261495 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp55t\" (UniqueName: \"kubernetes.io/projected/ca50db98-1579-42c1-8e66-1ebb6e7c5323-kube-api-access-mp55t\") pod \"maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq\" (UID: \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:00.263357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.263339 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh"] Oct 30 14:30:00.263592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.263575 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:00.271932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.271690 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k"] Oct 30 14:30:00.272309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.272156 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:00.282129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.281369 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:00.290837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.290818 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:30:00.290958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.290828 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9"] Oct 30 14:30:00.297946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.297929 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:00.298881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.298866 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz"] Oct 30 14:30:00.304020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.304002 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw"] Oct 30 14:30:00.315426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.315401 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh"] Oct 30 14:30:00.315531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.315441 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k"] Oct 30 14:30:00.316453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.315714 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:00.338200 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.338157 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09180ae1-ace5-43a7-9b01-635b157cdf4c" path="/var/lib/kubelet/pods/09180ae1-ace5-43a7-9b01-635b157cdf4c/volumes" Oct 30 14:30:00.338643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.338623 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9"] Oct 30 14:30:00.338744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.338678 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl"] Oct 30 14:30:00.347059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.347039 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs"] Oct 30 14:30:00.347209 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.347165 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:00.355423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.355404 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl"] Oct 30 14:30:00.355685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.355669 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw"] Oct 30 14:30:00.355870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.355856 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs"] Oct 30 14:30:00.356007 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.355616 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:00.363345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363321 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp55t\" (UniqueName: \"kubernetes.io/projected/ca50db98-1579-42c1-8e66-1ebb6e7c5323-kube-api-access-mp55t\") pod \"maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq\" (UID: \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:00.363451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363366 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k\" (UID: \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:00.363451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363414 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqsdk\" (UniqueName: \"kubernetes.io/projected/5ab9d46c-29a1-479a-a477-ae3f53404b91-kube-api-access-lqsdk\") pod \"maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz\" (UID: \"5ab9d46c-29a1-479a-a477-ae3f53404b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:00.363451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363447 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9d46c-29a1-479a-a477-ae3f53404b91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz\" (UID: \"5ab9d46c-29a1-479a-a477-ae3f53404b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:00.363626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363543 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlvm\" (UniqueName: \"kubernetes.io/projected/9a085f71-14d2-4210-b228-abd9eb70f463-kube-api-access-xqlvm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd\" (UID: \"9a085f71-14d2-4210-b228-abd9eb70f463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:00.363626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363594 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a225fb-a633-426a-8a53-f3214aa77efb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl\" (UID: \"b7a225fb-a633-426a-8a53-f3214aa77efb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:00.363727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363641 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78952e0-14c3-4811-8515-dc14aa6d7449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9\" (UID: \"b78952e0-14c3-4811-8515-dc14aa6d7449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:00.363727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363673 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8c9x\" (UniqueName: \"kubernetes.io/projected/5baa41cf-dc7b-4386-a579-92f33e4ae890-kube-api-access-n8c9x\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw\" (UID: \"5baa41cf-dc7b-4386-a579-92f33e4ae890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:00.363853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363735 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7wp\" (UniqueName: \"kubernetes.io/projected/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-kube-api-access-4j7wp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k\" (UID: \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:00.363853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363784 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfb9l\" (UniqueName: \"kubernetes.io/projected/b78952e0-14c3-4811-8515-dc14aa6d7449-kube-api-access-qfb9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9\" (UID: \"b78952e0-14c3-4811-8515-dc14aa6d7449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:00.363962 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363869 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1e643a-9a04-4a09-8d90-ccccdd02977c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh\" (UID: \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:00.363962 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363936 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl\" (UID: \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:00.364067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.363990 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7qtf\" (UniqueName: \"kubernetes.io/projected/4a1e643a-9a04-4a09-8d90-ccccdd02977c-kube-api-access-h7qtf\") pod \"maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh\" (UID: \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:00.364067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.364025 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd\" (UID: \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:00.364067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.364054 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baa41cf-dc7b-4386-a579-92f33e4ae890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw\" (UID: \"5baa41cf-dc7b-4386-a579-92f33e4ae890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:00.364267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.364086 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frk2v\" (UniqueName: \"kubernetes.io/projected/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-kube-api-access-frk2v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl\" (UID: \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:00.364267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.364150 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxgz\" (UniqueName: \"kubernetes.io/projected/b7a225fb-a633-426a-8a53-f3214aa77efb-kube-api-access-rpxgz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl\" (UID: \"b7a225fb-a633-426a-8a53-f3214aa77efb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:00.364384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.364270 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxc7f\" (UniqueName: \"kubernetes.io/projected/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-kube-api-access-mxc7f\") pod \"maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd\" (UID: \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:00.364450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.364387 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca50db98-1579-42c1-8e66-1ebb6e7c5323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq\" (UID: \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:00.364450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.364428 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a085f71-14d2-4210-b228-abd9eb70f463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd\" (UID: \"9a085f71-14d2-4210-b228-abd9eb70f463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:00.365047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.365018 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd\" (UID: \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:00.365183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.365162 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a085f71-14d2-4210-b228-abd9eb70f463-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd\" (UID: \"9a085f71-14d2-4210-b228-abd9eb70f463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:00.365733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.365707 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl\" (UID: \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:00.366138 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.366116 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca50db98-1579-42c1-8e66-1ebb6e7c5323-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq\" (UID: \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:00.391124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.391090 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp55t\" (UniqueName: \"kubernetes.io/projected/ca50db98-1579-42c1-8e66-1ebb6e7c5323-kube-api-access-mp55t\") pod \"maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq\" (UID: \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:00.393214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.393170 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlvm\" (UniqueName: \"kubernetes.io/projected/9a085f71-14d2-4210-b228-abd9eb70f463-kube-api-access-xqlvm\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd\" (UID: \"9a085f71-14d2-4210-b228-abd9eb70f463\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:00.394504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.394475 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frk2v\" (UniqueName: \"kubernetes.io/projected/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-kube-api-access-frk2v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl\" (UID: \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:00.396005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.395977 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxc7f\" (UniqueName: \"kubernetes.io/projected/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-kube-api-access-mxc7f\") pod \"maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd\" (UID: \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:00.467477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467440 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghpd4\" (UniqueName: \"kubernetes.io/projected/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-kube-api-access-ghpd4\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl\" (UID: \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:00.467622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467529 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k\" (UID: \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:00.467622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467587 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqsdk\" (UniqueName: \"kubernetes.io/projected/5ab9d46c-29a1-479a-a477-ae3f53404b91-kube-api-access-lqsdk\") pod \"maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz\" (UID: \"5ab9d46c-29a1-479a-a477-ae3f53404b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:00.467758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467628 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9d46c-29a1-479a-a477-ae3f53404b91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz\" (UID: \"5ab9d46c-29a1-479a-a477-ae3f53404b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:00.467758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467675 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a225fb-a633-426a-8a53-f3214aa77efb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl\" (UID: \"b7a225fb-a633-426a-8a53-f3214aa77efb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:00.467758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467717 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78952e0-14c3-4811-8515-dc14aa6d7449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9\" (UID: \"b78952e0-14c3-4811-8515-dc14aa6d7449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:00.467920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467758 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bf5cee-4198-421d-9b9f-633961ddba6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs\" (UID: \"59bf5cee-4198-421d-9b9f-633961ddba6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:00.467920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467797 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8c9x\" (UniqueName: \"kubernetes.io/projected/5baa41cf-dc7b-4386-a579-92f33e4ae890-kube-api-access-n8c9x\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw\" (UID: \"5baa41cf-dc7b-4386-a579-92f33e4ae890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:00.467920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467853 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl\" (UID: \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:00.467920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467893 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7wp\" (UniqueName: \"kubernetes.io/projected/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-kube-api-access-4j7wp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k\" (UID: \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:00.468225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467939 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfb9l\" (UniqueName: \"kubernetes.io/projected/b78952e0-14c3-4811-8515-dc14aa6d7449-kube-api-access-qfb9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9\" (UID: \"b78952e0-14c3-4811-8515-dc14aa6d7449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:00.468225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.467979 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1e643a-9a04-4a09-8d90-ccccdd02977c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh\" (UID: \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:00.468225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.468037 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7qtf\" (UniqueName: \"kubernetes.io/projected/4a1e643a-9a04-4a09-8d90-ccccdd02977c-kube-api-access-h7qtf\") pod \"maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh\" (UID: \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:00.468225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.468079 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baa41cf-dc7b-4386-a579-92f33e4ae890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw\" (UID: \"5baa41cf-dc7b-4386-a579-92f33e4ae890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:00.468225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.468148 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxgz\" (UniqueName: \"kubernetes.io/projected/b7a225fb-a633-426a-8a53-f3214aa77efb-kube-api-access-rpxgz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl\" (UID: \"b7a225fb-a633-426a-8a53-f3214aa77efb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:00.468483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.468308 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7v7\" (UniqueName: \"kubernetes.io/projected/59bf5cee-4198-421d-9b9f-633961ddba6d-kube-api-access-8h7v7\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs\" (UID: \"59bf5cee-4198-421d-9b9f-633961ddba6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:00.469257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.469234 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k\" (UID: \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:00.470326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.470297 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9d46c-29a1-479a-a477-ae3f53404b91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz\" (UID: \"5ab9d46c-29a1-479a-a477-ae3f53404b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:00.470853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.470826 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1e643a-9a04-4a09-8d90-ccccdd02977c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh\" (UID: \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:00.471083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.470978 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a225fb-a633-426a-8a53-f3214aa77efb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl\" (UID: \"b7a225fb-a633-426a-8a53-f3214aa77efb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:00.471303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.471079 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78952e0-14c3-4811-8515-dc14aa6d7449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9\" (UID: \"b78952e0-14c3-4811-8515-dc14aa6d7449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:00.471993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.471968 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baa41cf-dc7b-4386-a579-92f33e4ae890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw\" (UID: \"5baa41cf-dc7b-4386-a579-92f33e4ae890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:00.496157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.496114 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqsdk\" (UniqueName: \"kubernetes.io/projected/5ab9d46c-29a1-479a-a477-ae3f53404b91-kube-api-access-lqsdk\") pod \"maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz\" (UID: \"5ab9d46c-29a1-479a-a477-ae3f53404b91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:00.496490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.496465 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxgz\" (UniqueName: \"kubernetes.io/projected/b7a225fb-a633-426a-8a53-f3214aa77efb-kube-api-access-rpxgz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl\" (UID: \"b7a225fb-a633-426a-8a53-f3214aa77efb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:00.497742 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.497716 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8c9x\" (UniqueName: \"kubernetes.io/projected/5baa41cf-dc7b-4386-a579-92f33e4ae890-kube-api-access-n8c9x\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw\" (UID: \"5baa41cf-dc7b-4386-a579-92f33e4ae890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:00.497853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.497810 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7wp\" (UniqueName: \"kubernetes.io/projected/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-kube-api-access-4j7wp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k\" (UID: \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:00.498068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.498043 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfb9l\" (UniqueName: \"kubernetes.io/projected/b78952e0-14c3-4811-8515-dc14aa6d7449-kube-api-access-qfb9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9\" (UID: \"b78952e0-14c3-4811-8515-dc14aa6d7449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:00.498269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.498252 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7qtf\" (UniqueName: \"kubernetes.io/projected/4a1e643a-9a04-4a09-8d90-ccccdd02977c-kube-api-access-h7qtf\") pod \"maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh\" (UID: \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:00.528554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.528536 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:00.533615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.533598 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:00.570339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.570309 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bf5cee-4198-421d-9b9f-633961ddba6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs\" (UID: \"59bf5cee-4198-421d-9b9f-633961ddba6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:00.570490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.570373 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl\" (UID: \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:00.570629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.570607 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7v7\" (UniqueName: \"kubernetes.io/projected/59bf5cee-4198-421d-9b9f-633961ddba6d-kube-api-access-8h7v7\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs\" (UID: \"59bf5cee-4198-421d-9b9f-633961ddba6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:00.570721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.570704 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghpd4\" (UniqueName: \"kubernetes.io/projected/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-kube-api-access-ghpd4\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl\" (UID: \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:00.571139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.571119 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bf5cee-4198-421d-9b9f-633961ddba6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs\" (UID: \"59bf5cee-4198-421d-9b9f-633961ddba6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:00.571330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.571303 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl\" (UID: \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:00.588377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.586720 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:00.597805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.597744 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghpd4\" (UniqueName: \"kubernetes.io/projected/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-kube-api-access-ghpd4\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl\" (UID: \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:00.600182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.600154 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:00.608838 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.608814 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7v7\" (UniqueName: \"kubernetes.io/projected/59bf5cee-4198-421d-9b9f-633961ddba6d-kube-api-access-8h7v7\") pod \"maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs\" (UID: \"59bf5cee-4198-421d-9b9f-633961ddba6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:00.610425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.610372 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:00.635084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.635060 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:00.644277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.644257 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:00.655170 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.655147 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:00.671957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.671771 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:00.686726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.686703 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:00.689882 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.689388 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:00.695681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.695280 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:00.856307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.851006 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq"] Oct 30 14:30:00.884374 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.880471 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd"] Oct 30 14:30:00.892433 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:30:00.892399 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca50db98_1579_42c1_8e66_1ebb6e7c5323.slice/crio-57bec324adaeae251bb8992ebffb12f27ce4f282ed50a8b0ee359ef74df461c3 WatchSource:0}: Error finding container 57bec324adaeae251bb8992ebffb12f27ce4f282ed50a8b0ee359ef74df461c3: Status 404 returned error can't find the container with id 57bec324adaeae251bb8992ebffb12f27ce4f282ed50a8b0ee359ef74df461c3 Oct 30 14:30:00.992143 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:30:00.992108 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f21dc7_a231_42c9_a073_19ad3fb9f22b.slice/crio-54d7ee5a41261408bc38629c8d1975b4ee83863a2acff8e2827bfc8b9d5f88f3 WatchSource:0}: Error finding container 54d7ee5a41261408bc38629c8d1975b4ee83863a2acff8e2827bfc8b9d5f88f3: Status 404 returned error can't find the container with id 54d7ee5a41261408bc38629c8d1975b4ee83863a2acff8e2827bfc8b9d5f88f3 Oct 30 14:30:00.992556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:00.992504 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl"] Oct 30 14:30:01.237535 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:30:01.237502 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab9d46c_29a1_479a_a477_ae3f53404b91.slice/crio-5bae4f68a404796d7222df94def03eb329d34e2ff07a54f083a2d2d684a6041d WatchSource:0}: Error finding container 5bae4f68a404796d7222df94def03eb329d34e2ff07a54f083a2d2d684a6041d: Status 404 returned error can't find the container with id 5bae4f68a404796d7222df94def03eb329d34e2ff07a54f083a2d2d684a6041d Oct 30 14:30:01.238437 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:30:01.238412 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a225fb_a633_426a_8a53_f3214aa77efb.slice/crio-74d6171c2bf993e2574ed272d3832e553ab603c02047f722959b0deb25d2ced8 WatchSource:0}: Error finding container 74d6171c2bf993e2574ed272d3832e553ab603c02047f722959b0deb25d2ced8: Status 404 returned error can't find the container with id 74d6171c2bf993e2574ed272d3832e553ab603c02047f722959b0deb25d2ced8 Oct 30 14:30:01.239926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.239906 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz"] Oct 30 14:30:01.244828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.244686 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl"] Oct 30 14:30:01.258061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.257991 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd"] Oct 30 14:30:01.263682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.263656 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl"] Oct 30 14:30:01.267660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.267636 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw"] Oct 30 14:30:01.270255 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:30:01.270228 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1e643a_9a04_4a09_8d90_ccccdd02977c.slice/crio-775ff6c15c65dad1c3566b3297e621aa7534c265830ccced2bab2a6b25492f8e WatchSource:0}: Error finding container 775ff6c15c65dad1c3566b3297e621aa7534c265830ccced2bab2a6b25492f8e: Status 404 returned error can't find the container with id 775ff6c15c65dad1c3566b3297e621aa7534c265830ccced2bab2a6b25492f8e Oct 30 14:30:01.272136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.272016 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k"] Oct 30 14:30:01.275998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.275975 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh"] Oct 30 14:30:01.300493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.300466 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" event={"ID":"5baa41cf-dc7b-4386-a579-92f33e4ae890","Type":"ContainerStarted","Data":"a24209a67a93ded0b2d6ee3d65ee3bf1a7d5c09820f76304a60a68594ee664b9"} Oct 30 14:30:01.303107 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.303047 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" event={"ID":"a8f21dc7-a231-42c9-a073-19ad3fb9f22b","Type":"ContainerStarted","Data":"54d7ee5a41261408bc38629c8d1975b4ee83863a2acff8e2827bfc8b9d5f88f3"} Oct 30 14:30:01.304598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.304577 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" event={"ID":"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f","Type":"ContainerStarted","Data":"175e9f9909fd2dbf44d2002ca42a99f01c4853c3758a3dc94e84d97dcd512d94"} Oct 30 14:30:01.305927 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.305903 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" event={"ID":"ca50db98-1579-42c1-8e66-1ebb6e7c5323","Type":"ContainerStarted","Data":"57bec324adaeae251bb8992ebffb12f27ce4f282ed50a8b0ee359ef74df461c3"} Oct 30 14:30:01.307308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.307285 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" event={"ID":"5ab9d46c-29a1-479a-a477-ae3f53404b91","Type":"ContainerStarted","Data":"5bae4f68a404796d7222df94def03eb329d34e2ff07a54f083a2d2d684a6041d"} Oct 30 14:30:01.308614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.308590 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" event={"ID":"9a085f71-14d2-4210-b228-abd9eb70f463","Type":"ContainerStarted","Data":"7e9f4ca090cd8f93b2f0ab3d3102b0710e31fe0c471030fbb329ca40a711fb13"} Oct 30 14:30:01.309979 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.309958 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" event={"ID":"4a1e643a-9a04-4a09-8d90-ccccdd02977c","Type":"ContainerStarted","Data":"775ff6c15c65dad1c3566b3297e621aa7534c265830ccced2bab2a6b25492f8e"} Oct 30 14:30:01.311395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.311375 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" event={"ID":"b7a225fb-a633-426a-8a53-f3214aa77efb","Type":"ContainerStarted","Data":"74d6171c2bf993e2574ed272d3832e553ab603c02047f722959b0deb25d2ced8"} Oct 30 14:30:01.312873 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.312836 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" event={"ID":"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b","Type":"ContainerStarted","Data":"b4cd87cb724c3bfac0544ac73c6da82e98000ed13d93a7e1787d6d46c666d22d"} Oct 30 14:30:01.314509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.314490 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" event={"ID":"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d","Type":"ContainerStarted","Data":"76863803254c816a16968ded1fdecedc846adeaa36e3134866369e9d0c808e41"} Oct 30 14:30:01.336292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.336174 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs"] Oct 30 14:30:01.347395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:01.346993 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9"] Oct 30 14:30:01.549073 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:30:01.548971 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78952e0_14c3_4811_8515_dc14aa6d7449.slice/crio-11b05156c21a28e847340db7964f63d51cb31104787ba34dd536745d2b00acdf WatchSource:0}: Error finding container 11b05156c21a28e847340db7964f63d51cb31104787ba34dd536745d2b00acdf: Status 404 returned error can't find the container with id 11b05156c21a28e847340db7964f63d51cb31104787ba34dd536745d2b00acdf Oct 30 14:30:02.324420 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.324388 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" event={"ID":"5baa41cf-dc7b-4386-a579-92f33e4ae890","Type":"ContainerStarted","Data":"1f861bebfc7d6d1501676d4fd6e92a4fe5bde16cf2272de508002052c1e32b45"} Oct 30 14:30:02.326029 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.326000 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" event={"ID":"a8f21dc7-a231-42c9-a073-19ad3fb9f22b","Type":"ContainerStarted","Data":"3134f32b082dc4ab1be267b0bcc26e88e01271d9114f6a3ca30abec229661a08"} Oct 30 14:30:02.327727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.327702 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" event={"ID":"b78952e0-14c3-4811-8515-dc14aa6d7449","Type":"ContainerStarted","Data":"cca5936b338fec88e7222008dde0bbf355cce23f3331f960b16b3f229994daf9"} Oct 30 14:30:02.327828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.327751 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" event={"ID":"b78952e0-14c3-4811-8515-dc14aa6d7449","Type":"ContainerStarted","Data":"11b05156c21a28e847340db7964f63d51cb31104787ba34dd536745d2b00acdf"} Oct 30 14:30:02.329282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.329255 5824 generic.go:356] "Generic (PLEG): container finished" podID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerID="d8c16de3c9c4f75e75b8e6454d2a1d5a6fc48b557a126145bd957d9180191199" exitCode=60 Oct 30 14:30:02.329387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.329330 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" event={"ID":"ca50db98-1579-42c1-8e66-1ebb6e7c5323","Type":"ContainerDied","Data":"d8c16de3c9c4f75e75b8e6454d2a1d5a6fc48b557a126145bd957d9180191199"} Oct 30 14:30:02.329550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.329528 5824 scope.go:122] "RemoveContainer" containerID="d8c16de3c9c4f75e75b8e6454d2a1d5a6fc48b557a126145bd957d9180191199" Oct 30 14:30:02.331005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.330984 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" event={"ID":"5ab9d46c-29a1-479a-a477-ae3f53404b91","Type":"ContainerStarted","Data":"b20fa0d271677c2c9adab5fbf268efe10e1abe13d172b29baf4a8207aa135e53"} Oct 30 14:30:02.332877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.332854 5824 generic.go:356] "Generic (PLEG): container finished" podID="9a085f71-14d2-4210-b228-abd9eb70f463" containerID="63b029fa8441b930b19a22f305c65ac7bef2c697f5030ee67b0c78aee58796ff" exitCode=60 Oct 30 14:30:02.333000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.332956 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" event={"ID":"9a085f71-14d2-4210-b228-abd9eb70f463","Type":"ContainerDied","Data":"63b029fa8441b930b19a22f305c65ac7bef2c697f5030ee67b0c78aee58796ff"} Oct 30 14:30:02.333269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.333252 5824 scope.go:122] "RemoveContainer" containerID="63b029fa8441b930b19a22f305c65ac7bef2c697f5030ee67b0c78aee58796ff" Oct 30 14:30:02.335985 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.335951 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" event={"ID":"4a1e643a-9a04-4a09-8d90-ccccdd02977c","Type":"ContainerStarted","Data":"8ae7def23449a14299027e126193f7b55c272d2ba8c8361aab564472fc9d4d05"} Oct 30 14:30:02.339717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.339691 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" event={"ID":"59bf5cee-4198-421d-9b9f-633961ddba6d","Type":"ContainerStarted","Data":"763858af639183ab838c2f7b583eb831dc8ed3ef8ab597f8db54ee64f048290b"} Oct 30 14:30:02.351230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.348021 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" event={"ID":"b7a225fb-a633-426a-8a53-f3214aa77efb","Type":"ContainerStarted","Data":"7df24fc6a573a1171351a30eba120dbcbd63d1c52b517d471433aa275920d5ff"} Oct 30 14:30:02.356536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.356512 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" event={"ID":"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b","Type":"ContainerStarted","Data":"526b7b26ab18596839117b9bf8318430b7b79ffbb6a14ab1002cf59b57c10095"} Oct 30 14:30:02.358708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.358683 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" event={"ID":"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d","Type":"ContainerStarted","Data":"551c6f6c8f0741796a709cf48f60655a0b79a23a95174798a8e4b582f0374420"} Oct 30 14:30:02.375041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.374969 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" podStartSLOduration=2.374951507 podStartE2EDuration="2.374951507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.370638684 +0000 UTC m=+2432.171144130" watchObservedRunningTime="2026-10-30 14:30:02.374951507 +0000 UTC m=+2432.175456954" Oct 30 14:30:02.414653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.414584 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" podStartSLOduration=2.41456962 podStartE2EDuration="2.41456962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.413240328 +0000 UTC m=+2432.213745773" watchObservedRunningTime="2026-10-30 14:30:02.41456962 +0000 UTC m=+2432.215075064" Oct 30 14:30:02.463812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.463753 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" podStartSLOduration=2.463734389 podStartE2EDuration="2.463734389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.458451042 +0000 UTC m=+2432.258956487" watchObservedRunningTime="2026-10-30 14:30:02.463734389 +0000 UTC m=+2432.264239835" Oct 30 14:30:02.559216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.557480 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" podStartSLOduration=2.557462996 podStartE2EDuration="2.557462996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.551525673 +0000 UTC m=+2432.352031121" watchObservedRunningTime="2026-10-30 14:30:02.557462996 +0000 UTC m=+2432.357968441" Oct 30 14:30:02.596252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.596180 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" podStartSLOduration=2.596165296 podStartE2EDuration="2.596165296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.595162987 +0000 UTC m=+2432.395668432" watchObservedRunningTime="2026-10-30 14:30:02.596165296 +0000 UTC m=+2432.396670740" Oct 30 14:30:02.685544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.685494 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" podStartSLOduration=2.6854775220000002 podStartE2EDuration="2.685477522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.683578841 +0000 UTC m=+2432.484084286" watchObservedRunningTime="2026-10-30 14:30:02.685477522 +0000 UTC m=+2432.485982999" Oct 30 14:30:02.731316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.731262 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" podStartSLOduration=2.731243848 podStartE2EDuration="2.731243848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.727142548 +0000 UTC m=+2432.527647989" watchObservedRunningTime="2026-10-30 14:30:02.731243848 +0000 UTC m=+2432.531749295" Oct 30 14:30:02.773360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:02.773314 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" podStartSLOduration=2.773299263 podStartE2EDuration="2.773299263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:02.772946813 +0000 UTC m=+2432.573452268" watchObservedRunningTime="2026-10-30 14:30:02.773299263 +0000 UTC m=+2432.573804706" Oct 30 14:30:03.368322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.368278 5824 generic.go:356] "Generic (PLEG): container finished" podID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerID="fe21217666d0e062fb9d983a3aa00b337a03da9463d2546c26ee80e2ab441912" exitCode=60 Oct 30 14:30:03.368685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.368322 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" event={"ID":"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f","Type":"ContainerDied","Data":"fe21217666d0e062fb9d983a3aa00b337a03da9463d2546c26ee80e2ab441912"} Oct 30 14:30:03.368685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.368631 5824 scope.go:122] "RemoveContainer" containerID="fe21217666d0e062fb9d983a3aa00b337a03da9463d2546c26ee80e2ab441912" Oct 30 14:30:03.370247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.370225 5824 generic.go:356] "Generic (PLEG): container finished" podID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerID="cca5936b338fec88e7222008dde0bbf355cce23f3331f960b16b3f229994daf9" exitCode=60 Oct 30 14:30:03.370348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.370308 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" event={"ID":"b78952e0-14c3-4811-8515-dc14aa6d7449","Type":"ContainerDied","Data":"cca5936b338fec88e7222008dde0bbf355cce23f3331f960b16b3f229994daf9"} Oct 30 14:30:03.370603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.370585 5824 scope.go:122] "RemoveContainer" containerID="cca5936b338fec88e7222008dde0bbf355cce23f3331f960b16b3f229994daf9" Oct 30 14:30:03.372557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.372535 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" event={"ID":"ca50db98-1579-42c1-8e66-1ebb6e7c5323","Type":"ContainerStarted","Data":"fc8171c4dce2ec1dbc6eab69fb9fd044a91c0731e5e84c61de5fb7cfe6dfd57f"} Oct 30 14:30:03.374260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.374241 5824 generic.go:356] "Generic (PLEG): container finished" podID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerID="b20fa0d271677c2c9adab5fbf268efe10e1abe13d172b29baf4a8207aa135e53" exitCode=60 Oct 30 14:30:03.374339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.374297 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" event={"ID":"5ab9d46c-29a1-479a-a477-ae3f53404b91","Type":"ContainerDied","Data":"b20fa0d271677c2c9adab5fbf268efe10e1abe13d172b29baf4a8207aa135e53"} Oct 30 14:30:03.374548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.374526 5824 scope.go:122] "RemoveContainer" containerID="b20fa0d271677c2c9adab5fbf268efe10e1abe13d172b29baf4a8207aa135e53" Oct 30 14:30:03.376437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.376415 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" event={"ID":"9a085f71-14d2-4210-b228-abd9eb70f463","Type":"ContainerStarted","Data":"fba75e3085df0221f46f6a8dbdc2dbb62f3ac887f8a2742fe3d9f03444ee2dd7"} Oct 30 14:30:03.378007 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.377983 5824 generic.go:356] "Generic (PLEG): container finished" podID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerID="8ae7def23449a14299027e126193f7b55c272d2ba8c8361aab564472fc9d4d05" exitCode=60 Oct 30 14:30:03.378092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.378044 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" event={"ID":"4a1e643a-9a04-4a09-8d90-ccccdd02977c","Type":"ContainerDied","Data":"8ae7def23449a14299027e126193f7b55c272d2ba8c8361aab564472fc9d4d05"} Oct 30 14:30:03.378272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.378257 5824 scope.go:122] "RemoveContainer" containerID="8ae7def23449a14299027e126193f7b55c272d2ba8c8361aab564472fc9d4d05" Oct 30 14:30:03.379578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.379556 5824 generic.go:356] "Generic (PLEG): container finished" podID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerID="2ab0d6b77530a59d18b3f06a49aeb64f93765aef8855e282fd9e809eb755c236" exitCode=60 Oct 30 14:30:03.379674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.379628 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" event={"ID":"59bf5cee-4198-421d-9b9f-633961ddba6d","Type":"ContainerDied","Data":"2ab0d6b77530a59d18b3f06a49aeb64f93765aef8855e282fd9e809eb755c236"} Oct 30 14:30:03.379830 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.379814 5824 scope.go:122] "RemoveContainer" containerID="2ab0d6b77530a59d18b3f06a49aeb64f93765aef8855e282fd9e809eb755c236" Oct 30 14:30:03.381915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.381895 5824 generic.go:356] "Generic (PLEG): container finished" podID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerID="7df24fc6a573a1171351a30eba120dbcbd63d1c52b517d471433aa275920d5ff" exitCode=60 Oct 30 14:30:03.381999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.381967 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" event={"ID":"b7a225fb-a633-426a-8a53-f3214aa77efb","Type":"ContainerDied","Data":"7df24fc6a573a1171351a30eba120dbcbd63d1c52b517d471433aa275920d5ff"} Oct 30 14:30:03.382162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.382147 5824 scope.go:122] "RemoveContainer" containerID="7df24fc6a573a1171351a30eba120dbcbd63d1c52b517d471433aa275920d5ff" Oct 30 14:30:03.383909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.383887 5824 generic.go:356] "Generic (PLEG): container finished" podID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerID="526b7b26ab18596839117b9bf8318430b7b79ffbb6a14ab1002cf59b57c10095" exitCode=60 Oct 30 14:30:03.383991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.383936 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" event={"ID":"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b","Type":"ContainerDied","Data":"526b7b26ab18596839117b9bf8318430b7b79ffbb6a14ab1002cf59b57c10095"} Oct 30 14:30:03.384178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.384164 5824 scope.go:122] "RemoveContainer" containerID="526b7b26ab18596839117b9bf8318430b7b79ffbb6a14ab1002cf59b57c10095" Oct 30 14:30:03.385913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.385891 5824 generic.go:356] "Generic (PLEG): container finished" podID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerID="551c6f6c8f0741796a709cf48f60655a0b79a23a95174798a8e4b582f0374420" exitCode=60 Oct 30 14:30:03.386011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.385924 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" event={"ID":"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d","Type":"ContainerDied","Data":"551c6f6c8f0741796a709cf48f60655a0b79a23a95174798a8e4b582f0374420"} Oct 30 14:30:03.386218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.386171 5824 scope.go:122] "RemoveContainer" containerID="551c6f6c8f0741796a709cf48f60655a0b79a23a95174798a8e4b582f0374420" Oct 30 14:30:03.388048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.388026 5824 generic.go:356] "Generic (PLEG): container finished" podID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerID="1f861bebfc7d6d1501676d4fd6e92a4fe5bde16cf2272de508002052c1e32b45" exitCode=60 Oct 30 14:30:03.388134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.388098 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" event={"ID":"5baa41cf-dc7b-4386-a579-92f33e4ae890","Type":"ContainerDied","Data":"1f861bebfc7d6d1501676d4fd6e92a4fe5bde16cf2272de508002052c1e32b45"} Oct 30 14:30:03.388335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.388318 5824 scope.go:122] "RemoveContainer" containerID="1f861bebfc7d6d1501676d4fd6e92a4fe5bde16cf2272de508002052c1e32b45" Oct 30 14:30:03.391784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.391050 5824 generic.go:356] "Generic (PLEG): container finished" podID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerID="3134f32b082dc4ab1be267b0bcc26e88e01271d9114f6a3ca30abec229661a08" exitCode=60 Oct 30 14:30:03.391784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.391082 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" event={"ID":"a8f21dc7-a231-42c9-a073-19ad3fb9f22b","Type":"ContainerDied","Data":"3134f32b082dc4ab1be267b0bcc26e88e01271d9114f6a3ca30abec229661a08"} Oct 30 14:30:03.391784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.391741 5824 scope.go:122] "RemoveContainer" containerID="3134f32b082dc4ab1be267b0bcc26e88e01271d9114f6a3ca30abec229661a08" Oct 30 14:30:03.507714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.507669 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" podStartSLOduration=3.507657012 podStartE2EDuration="3.507657012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:03.503594042 +0000 UTC m=+2433.304099492" watchObservedRunningTime="2026-10-30 14:30:03.507657012 +0000 UTC m=+2433.308162455" Oct 30 14:30:03.685933 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:03.685906 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:30:03.934614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:03.934563 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" podStartSLOduration=3.9345474129999998 podStartE2EDuration="3.934547413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:03.885451991 +0000 UTC m=+2433.685957436" watchObservedRunningTime="2026-10-30 14:30:03.934547413 +0000 UTC m=+2433.735052857" Oct 30 14:30:04.402080 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.402004 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" event={"ID":"b7a225fb-a633-426a-8a53-f3214aa77efb","Type":"ContainerStarted","Data":"d217dd5d95f2f5519a2b27c5c6188ac4f0e89daf1faade31eee041cda9eae8c5"} Oct 30 14:30:04.404312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.404283 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" event={"ID":"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b","Type":"ContainerStarted","Data":"561792a5d8e95e2cfc8491e7d3b6cb20067944775c19c0753b6632856ccfe920"} Oct 30 14:30:04.406284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.406260 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" event={"ID":"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d","Type":"ContainerStarted","Data":"5eef11ab9b3b88c0ff5645bd7e12f12c899a1a4dada49eaa62dd71696451c02a"} Oct 30 14:30:04.408181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.408158 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" event={"ID":"5baa41cf-dc7b-4386-a579-92f33e4ae890","Type":"ContainerStarted","Data":"1a7a55e4a34a39fcda37d15034313008b90b4cc42dd3d7b8f3c0b828d6fb3b25"} Oct 30 14:30:04.410071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.410046 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" event={"ID":"a8f21dc7-a231-42c9-a073-19ad3fb9f22b","Type":"ContainerStarted","Data":"5abae36e5409c58300f460761d629c761efcdef1c9c7c8ca7d7c5aeab8a16839"} Oct 30 14:30:04.411988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.411967 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" event={"ID":"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f","Type":"ContainerStarted","Data":"8a5c2db06132e5f1fe458457f83930ee3612deed7c8c5b117fde46b6f3fb3918"} Oct 30 14:30:04.413978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.413954 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" event={"ID":"b78952e0-14c3-4811-8515-dc14aa6d7449","Type":"ContainerStarted","Data":"d44022c0e54d14f3782d009a8209b519f5be4127984e527f97d2a22370b6850a"} Oct 30 14:30:04.415861 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.415832 5824 generic.go:356] "Generic (PLEG): container finished" podID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerID="fc8171c4dce2ec1dbc6eab69fb9fd044a91c0731e5e84c61de5fb7cfe6dfd57f" exitCode=60 Oct 30 14:30:04.415958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.415916 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" event={"ID":"ca50db98-1579-42c1-8e66-1ebb6e7c5323","Type":"ContainerDied","Data":"fc8171c4dce2ec1dbc6eab69fb9fd044a91c0731e5e84c61de5fb7cfe6dfd57f"} Oct 30 14:30:04.416331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.415975 5824 scope.go:122] "RemoveContainer" containerID="d8c16de3c9c4f75e75b8e6454d2a1d5a6fc48b557a126145bd957d9180191199" Oct 30 14:30:04.416331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.416106 5824 scope.go:122] "RemoveContainer" containerID="fc8171c4dce2ec1dbc6eab69fb9fd044a91c0731e5e84c61de5fb7cfe6dfd57f" Oct 30 14:30:04.416454 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:04.416384 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq_odh-ai-gateway-infra(ca50db98-1579-42c1-8e66-1ebb6e7c5323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" Oct 30 14:30:04.418149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.418126 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" event={"ID":"5ab9d46c-29a1-479a-a477-ae3f53404b91","Type":"ContainerStarted","Data":"19cd0d7e80cf1dd844db19723559cfe4b95b0ed9b1a7e461d470a4116635a0c5"} Oct 30 14:30:04.420228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.420207 5824 generic.go:356] "Generic (PLEG): container finished" podID="9a085f71-14d2-4210-b228-abd9eb70f463" containerID="fba75e3085df0221f46f6a8dbdc2dbb62f3ac887f8a2742fe3d9f03444ee2dd7" exitCode=60 Oct 30 14:30:04.420333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.420313 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" event={"ID":"9a085f71-14d2-4210-b228-abd9eb70f463","Type":"ContainerDied","Data":"fba75e3085df0221f46f6a8dbdc2dbb62f3ac887f8a2742fe3d9f03444ee2dd7"} Oct 30 14:30:04.420518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.420499 5824 scope.go:122] "RemoveContainer" containerID="fba75e3085df0221f46f6a8dbdc2dbb62f3ac887f8a2742fe3d9f03444ee2dd7" Oct 30 14:30:04.420760 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:04.420738 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd_odh-ai-gateway-infra(9a085f71-14d2-4210-b228-abd9eb70f463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" Oct 30 14:30:04.422569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.422545 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" event={"ID":"4a1e643a-9a04-4a09-8d90-ccccdd02977c","Type":"ContainerStarted","Data":"ecf8c1756a50e7798f003a44df92196d641229b65456a5699df130ce776351d4"} Oct 30 14:30:04.424567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.424544 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" event={"ID":"59bf5cee-4198-421d-9b9f-633961ddba6d","Type":"ContainerStarted","Data":"d9f40356054d0e44d6d7f55a7609490f07bf42e977ddf5c86c6810c17799cf08"} Oct 30 14:30:04.434110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.434092 5824 scope.go:122] "RemoveContainer" containerID="63b029fa8441b930b19a22f305c65ac7bef2c697f5030ee67b0c78aee58796ff" Oct 30 14:30:04.670463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.670358 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" podStartSLOduration=4.670340951 podStartE2EDuration="4.670340951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:04.666231344 +0000 UTC m=+2434.466736788" watchObservedRunningTime="2026-10-30 14:30:04.670340951 +0000 UTC m=+2434.470846399" Oct 30 14:30:04.973499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:04.973455 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" podStartSLOduration=4.973440544 podStartE2EDuration="4.973440544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:04.972049831 +0000 UTC m=+2434.772555276" watchObservedRunningTime="2026-10-30 14:30:04.973440544 +0000 UTC m=+2434.773945987" Oct 30 14:30:05.435474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.435441 5824 generic.go:356] "Generic (PLEG): container finished" podID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerID="8a5c2db06132e5f1fe458457f83930ee3612deed7c8c5b117fde46b6f3fb3918" exitCode=60 Oct 30 14:30:05.435902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.435520 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" event={"ID":"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f","Type":"ContainerDied","Data":"8a5c2db06132e5f1fe458457f83930ee3612deed7c8c5b117fde46b6f3fb3918"} Oct 30 14:30:05.435902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.435580 5824 scope.go:122] "RemoveContainer" containerID="fe21217666d0e062fb9d983a3aa00b337a03da9463d2546c26ee80e2ab441912" Oct 30 14:30:05.435902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.435825 5824 scope.go:122] "RemoveContainer" containerID="8a5c2db06132e5f1fe458457f83930ee3612deed7c8c5b117fde46b6f3fb3918" Oct 30 14:30:05.436144 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.436119 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl_odh-ai-gateway-infra(cadd310c-c30e-4ad2-b5b3-2af8623e7e3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" Oct 30 14:30:05.437743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.437717 5824 generic.go:356] "Generic (PLEG): container finished" podID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerID="d44022c0e54d14f3782d009a8209b519f5be4127984e527f97d2a22370b6850a" exitCode=60 Oct 30 14:30:05.437859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.437791 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" event={"ID":"b78952e0-14c3-4811-8515-dc14aa6d7449","Type":"ContainerDied","Data":"d44022c0e54d14f3782d009a8209b519f5be4127984e527f97d2a22370b6850a"} Oct 30 14:30:05.438007 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.437992 5824 scope.go:122] "RemoveContainer" containerID="d44022c0e54d14f3782d009a8209b519f5be4127984e527f97d2a22370b6850a" Oct 30 14:30:05.438271 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.438249 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9_odh-ai-gateway-infra(b78952e0-14c3-4811-8515-dc14aa6d7449)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" Oct 30 14:30:05.439757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.439732 5824 scope.go:122] "RemoveContainer" containerID="fc8171c4dce2ec1dbc6eab69fb9fd044a91c0731e5e84c61de5fb7cfe6dfd57f" Oct 30 14:30:05.439996 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.439975 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq_odh-ai-gateway-infra(ca50db98-1579-42c1-8e66-1ebb6e7c5323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" Oct 30 14:30:05.441448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.441427 5824 generic.go:356] "Generic (PLEG): container finished" podID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerID="19cd0d7e80cf1dd844db19723559cfe4b95b0ed9b1a7e461d470a4116635a0c5" exitCode=60 Oct 30 14:30:05.441553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.441500 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" event={"ID":"5ab9d46c-29a1-479a-a477-ae3f53404b91","Type":"ContainerDied","Data":"19cd0d7e80cf1dd844db19723559cfe4b95b0ed9b1a7e461d470a4116635a0c5"} Oct 30 14:30:05.441693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.441678 5824 scope.go:122] "RemoveContainer" containerID="19cd0d7e80cf1dd844db19723559cfe4b95b0ed9b1a7e461d470a4116635a0c5" Oct 30 14:30:05.441912 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.441894 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz_odh-ai-gateway-infra(5ab9d46c-29a1-479a-a477-ae3f53404b91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" Oct 30 14:30:05.443394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.443378 5824 scope.go:122] "RemoveContainer" containerID="fba75e3085df0221f46f6a8dbdc2dbb62f3ac887f8a2742fe3d9f03444ee2dd7" Oct 30 14:30:05.443611 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.443590 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd_odh-ai-gateway-infra(9a085f71-14d2-4210-b228-abd9eb70f463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" Oct 30 14:30:05.445117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.445095 5824 generic.go:356] "Generic (PLEG): container finished" podID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerID="ecf8c1756a50e7798f003a44df92196d641229b65456a5699df130ce776351d4" exitCode=60 Oct 30 14:30:05.445238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.445174 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" event={"ID":"4a1e643a-9a04-4a09-8d90-ccccdd02977c","Type":"ContainerDied","Data":"ecf8c1756a50e7798f003a44df92196d641229b65456a5699df130ce776351d4"} Oct 30 14:30:05.445388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.445369 5824 scope.go:122] "RemoveContainer" containerID="ecf8c1756a50e7798f003a44df92196d641229b65456a5699df130ce776351d4" Oct 30 14:30:05.445651 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.445620 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh_odh-ai-gateway-infra(4a1e643a-9a04-4a09-8d90-ccccdd02977c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" Oct 30 14:30:05.447344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.447321 5824 generic.go:356] "Generic (PLEG): container finished" podID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerID="d9f40356054d0e44d6d7f55a7609490f07bf42e977ddf5c86c6810c17799cf08" exitCode=60 Oct 30 14:30:05.447430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.447416 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" event={"ID":"59bf5cee-4198-421d-9b9f-633961ddba6d","Type":"ContainerDied","Data":"d9f40356054d0e44d6d7f55a7609490f07bf42e977ddf5c86c6810c17799cf08"} Oct 30 14:30:05.447685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.447653 5824 scope.go:122] "RemoveContainer" containerID="d9f40356054d0e44d6d7f55a7609490f07bf42e977ddf5c86c6810c17799cf08" Oct 30 14:30:05.447946 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.447921 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs_odh-ai-gateway-infra(59bf5cee-4198-421d-9b9f-633961ddba6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" Oct 30 14:30:05.449289 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.449263 5824 generic.go:356] "Generic (PLEG): container finished" podID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerID="d217dd5d95f2f5519a2b27c5c6188ac4f0e89daf1faade31eee041cda9eae8c5" exitCode=60 Oct 30 14:30:05.449391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.449340 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" event={"ID":"b7a225fb-a633-426a-8a53-f3214aa77efb","Type":"ContainerDied","Data":"d217dd5d95f2f5519a2b27c5c6188ac4f0e89daf1faade31eee041cda9eae8c5"} Oct 30 14:30:05.449547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.449530 5824 scope.go:122] "RemoveContainer" containerID="d217dd5d95f2f5519a2b27c5c6188ac4f0e89daf1faade31eee041cda9eae8c5" Oct 30 14:30:05.449793 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.449773 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl_odh-ai-gateway-infra(b7a225fb-a633-426a-8a53-f3214aa77efb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" Oct 30 14:30:05.454327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.454306 5824 scope.go:122] "RemoveContainer" containerID="cca5936b338fec88e7222008dde0bbf355cce23f3331f960b16b3f229994daf9" Oct 30 14:30:05.455633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.455615 5824 generic.go:356] "Generic (PLEG): container finished" podID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerID="561792a5d8e95e2cfc8491e7d3b6cb20067944775c19c0753b6632856ccfe920" exitCode=60 Oct 30 14:30:05.455726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.455677 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" event={"ID":"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b","Type":"ContainerDied","Data":"561792a5d8e95e2cfc8491e7d3b6cb20067944775c19c0753b6632856ccfe920"} Oct 30 14:30:05.455904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.455881 5824 scope.go:122] "RemoveContainer" containerID="561792a5d8e95e2cfc8491e7d3b6cb20067944775c19c0753b6632856ccfe920" Oct 30 14:30:05.456150 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.456129 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k_odh-ai-gateway-infra(dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" Oct 30 14:30:05.457618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.457599 5824 generic.go:356] "Generic (PLEG): container finished" podID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerID="5eef11ab9b3b88c0ff5645bd7e12f12c899a1a4dada49eaa62dd71696451c02a" exitCode=60 Oct 30 14:30:05.457705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.457680 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" event={"ID":"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d","Type":"ContainerDied","Data":"5eef11ab9b3b88c0ff5645bd7e12f12c899a1a4dada49eaa62dd71696451c02a"} Oct 30 14:30:05.457871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.457854 5824 scope.go:122] "RemoveContainer" containerID="5eef11ab9b3b88c0ff5645bd7e12f12c899a1a4dada49eaa62dd71696451c02a" Oct 30 14:30:05.458127 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.458104 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd_odh-ai-gateway-infra(ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" Oct 30 14:30:05.459555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.459535 5824 generic.go:356] "Generic (PLEG): container finished" podID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerID="1a7a55e4a34a39fcda37d15034313008b90b4cc42dd3d7b8f3c0b828d6fb3b25" exitCode=60 Oct 30 14:30:05.459650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.459614 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" event={"ID":"5baa41cf-dc7b-4386-a579-92f33e4ae890","Type":"ContainerDied","Data":"1a7a55e4a34a39fcda37d15034313008b90b4cc42dd3d7b8f3c0b828d6fb3b25"} Oct 30 14:30:05.459783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.459769 5824 scope.go:122] "RemoveContainer" containerID="1a7a55e4a34a39fcda37d15034313008b90b4cc42dd3d7b8f3c0b828d6fb3b25" Oct 30 14:30:05.460011 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.459986 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw_odh-ai-gateway-infra(5baa41cf-dc7b-4386-a579-92f33e4ae890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" Oct 30 14:30:05.461566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.461549 5824 generic.go:356] "Generic (PLEG): container finished" podID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerID="5abae36e5409c58300f460761d629c761efcdef1c9c7c8ca7d7c5aeab8a16839" exitCode=60 Oct 30 14:30:05.461650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.461575 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" event={"ID":"a8f21dc7-a231-42c9-a073-19ad3fb9f22b","Type":"ContainerDied","Data":"5abae36e5409c58300f460761d629c761efcdef1c9c7c8ca7d7c5aeab8a16839"} Oct 30 14:30:05.461793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.461779 5824 scope.go:122] "RemoveContainer" containerID="5abae36e5409c58300f460761d629c761efcdef1c9c7c8ca7d7c5aeab8a16839" Oct 30 14:30:05.461991 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:05.461975 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl_odh-ai-gateway-infra(a8f21dc7-a231-42c9-a073-19ad3fb9f22b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" Oct 30 14:30:05.469637 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.469618 5824 scope.go:122] "RemoveContainer" containerID="b20fa0d271677c2c9adab5fbf268efe10e1abe13d172b29baf4a8207aa135e53" Oct 30 14:30:05.490774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.490751 5824 scope.go:122] "RemoveContainer" containerID="8ae7def23449a14299027e126193f7b55c272d2ba8c8361aab564472fc9d4d05" Oct 30 14:30:05.508097 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.508078 5824 scope.go:122] "RemoveContainer" containerID="2ab0d6b77530a59d18b3f06a49aeb64f93765aef8855e282fd9e809eb755c236" Oct 30 14:30:05.526060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.526036 5824 scope.go:122] "RemoveContainer" containerID="7df24fc6a573a1171351a30eba120dbcbd63d1c52b517d471433aa275920d5ff" Oct 30 14:30:05.540480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.540461 5824 scope.go:122] "RemoveContainer" containerID="526b7b26ab18596839117b9bf8318430b7b79ffbb6a14ab1002cf59b57c10095" Oct 30 14:30:05.553747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.553729 5824 scope.go:122] "RemoveContainer" containerID="551c6f6c8f0741796a709cf48f60655a0b79a23a95174798a8e4b582f0374420" Oct 30 14:30:05.566852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.566834 5824 scope.go:122] "RemoveContainer" containerID="1f861bebfc7d6d1501676d4fd6e92a4fe5bde16cf2272de508002052c1e32b45" Oct 30 14:30:05.581124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:05.581109 5824 scope.go:122] "RemoveContainer" containerID="3134f32b082dc4ab1be267b0bcc26e88e01271d9114f6a3ca30abec229661a08" Oct 30 14:30:06.470628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.470602 5824 scope.go:122] "RemoveContainer" containerID="1a7a55e4a34a39fcda37d15034313008b90b4cc42dd3d7b8f3c0b828d6fb3b25" Oct 30 14:30:06.471017 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.470909 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw_odh-ai-gateway-infra(5baa41cf-dc7b-4386-a579-92f33e4ae890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" Oct 30 14:30:06.472319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.472297 5824 scope.go:122] "RemoveContainer" containerID="5abae36e5409c58300f460761d629c761efcdef1c9c7c8ca7d7c5aeab8a16839" Oct 30 14:30:06.472545 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.472525 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl_odh-ai-gateway-infra(a8f21dc7-a231-42c9-a073-19ad3fb9f22b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" Oct 30 14:30:06.473936 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.473914 5824 scope.go:122] "RemoveContainer" containerID="8a5c2db06132e5f1fe458457f83930ee3612deed7c8c5b117fde46b6f3fb3918" Oct 30 14:30:06.474200 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.474170 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl_odh-ai-gateway-infra(cadd310c-c30e-4ad2-b5b3-2af8623e7e3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" Oct 30 14:30:06.481418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.481393 5824 scope.go:122] "RemoveContainer" containerID="d44022c0e54d14f3782d009a8209b519f5be4127984e527f97d2a22370b6850a" Oct 30 14:30:06.481597 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.481578 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9_odh-ai-gateway-infra(b78952e0-14c3-4811-8515-dc14aa6d7449)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" Oct 30 14:30:06.484119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.484099 5824 scope.go:122] "RemoveContainer" containerID="19cd0d7e80cf1dd844db19723559cfe4b95b0ed9b1a7e461d470a4116635a0c5" Oct 30 14:30:06.484403 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.484378 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz_odh-ai-gateway-infra(5ab9d46c-29a1-479a-a477-ae3f53404b91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" Oct 30 14:30:06.485885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.485864 5824 scope.go:122] "RemoveContainer" containerID="ecf8c1756a50e7798f003a44df92196d641229b65456a5699df130ce776351d4" Oct 30 14:30:06.486138 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.486116 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh_odh-ai-gateway-infra(4a1e643a-9a04-4a09-8d90-ccccdd02977c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" Oct 30 14:30:06.487997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.487975 5824 scope.go:122] "RemoveContainer" containerID="d9f40356054d0e44d6d7f55a7609490f07bf42e977ddf5c86c6810c17799cf08" Oct 30 14:30:06.488282 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.488258 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs_odh-ai-gateway-infra(59bf5cee-4198-421d-9b9f-633961ddba6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" Oct 30 14:30:06.489652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.489633 5824 scope.go:122] "RemoveContainer" containerID="d217dd5d95f2f5519a2b27c5c6188ac4f0e89daf1faade31eee041cda9eae8c5" Oct 30 14:30:06.489856 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.489834 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl_odh-ai-gateway-infra(b7a225fb-a633-426a-8a53-f3214aa77efb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" Oct 30 14:30:06.491219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.491180 5824 scope.go:122] "RemoveContainer" containerID="561792a5d8e95e2cfc8491e7d3b6cb20067944775c19c0753b6632856ccfe920" Oct 30 14:30:06.491415 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.491394 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k_odh-ai-gateway-infra(dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" Oct 30 14:30:06.492661 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:06.492640 5824 scope.go:122] "RemoveContainer" containerID="5eef11ab9b3b88c0ff5645bd7e12f12c899a1a4dada49eaa62dd71696451c02a" Oct 30 14:30:06.492863 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:06.492844 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd_odh-ai-gateway-infra(ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" Oct 30 14:30:15.309648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:15.309563 5824 scope.go:122] "RemoveContainer" containerID="561792a5d8e95e2cfc8491e7d3b6cb20067944775c19c0753b6632856ccfe920" Oct 30 14:30:15.310267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:15.309946 5824 scope.go:122] "RemoveContainer" containerID="ecf8c1756a50e7798f003a44df92196d641229b65456a5699df130ce776351d4" Oct 30 14:30:16.309685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.309658 5824 scope.go:122] "RemoveContainer" containerID="8a5c2db06132e5f1fe458457f83930ee3612deed7c8c5b117fde46b6f3fb3918" Oct 30 14:30:16.310035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.309852 5824 scope.go:122] "RemoveContainer" containerID="5abae36e5409c58300f460761d629c761efcdef1c9c7c8ca7d7c5aeab8a16839" Oct 30 14:30:16.310035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.309953 5824 scope.go:122] "RemoveContainer" containerID="d44022c0e54d14f3782d009a8209b519f5be4127984e527f97d2a22370b6850a" Oct 30 14:30:16.310445 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.310369 5824 scope.go:122] "RemoveContainer" containerID="fba75e3085df0221f46f6a8dbdc2dbb62f3ac887f8a2742fe3d9f03444ee2dd7" Oct 30 14:30:16.578073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.578045 5824 generic.go:356] "Generic (PLEG): container finished" podID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerID="26949573c451e056822082059879023086383abb5b7e282e38470d791e9f0025" exitCode=60 Oct 30 14:30:16.578220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.578116 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" event={"ID":"4a1e643a-9a04-4a09-8d90-ccccdd02977c","Type":"ContainerDied","Data":"26949573c451e056822082059879023086383abb5b7e282e38470d791e9f0025"} Oct 30 14:30:16.578220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.578175 5824 scope.go:122] "RemoveContainer" containerID="ecf8c1756a50e7798f003a44df92196d641229b65456a5699df130ce776351d4" Oct 30 14:30:16.578683 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.578658 5824 scope.go:122] "RemoveContainer" containerID="26949573c451e056822082059879023086383abb5b7e282e38470d791e9f0025" Oct 30 14:30:16.579136 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:16.579106 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh_odh-ai-gateway-infra(4a1e643a-9a04-4a09-8d90-ccccdd02977c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" Oct 30 14:30:16.580588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.580564 5824 generic.go:356] "Generic (PLEG): container finished" podID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerID="39ba5132852b71a44353386d19707154485e2a45a4ecd63a110851bc87cd7fb2" exitCode=60 Oct 30 14:30:16.580674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.580641 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" event={"ID":"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b","Type":"ContainerDied","Data":"39ba5132852b71a44353386d19707154485e2a45a4ecd63a110851bc87cd7fb2"} Oct 30 14:30:16.581076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.581061 5824 scope.go:122] "RemoveContainer" containerID="39ba5132852b71a44353386d19707154485e2a45a4ecd63a110851bc87cd7fb2" Oct 30 14:30:16.581418 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:16.581399 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k_odh-ai-gateway-infra(dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" Oct 30 14:30:16.591720 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:16.591703 5824 scope.go:122] "RemoveContainer" containerID="561792a5d8e95e2cfc8491e7d3b6cb20067944775c19c0753b6632856ccfe920" Oct 30 14:30:17.309326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.309296 5824 scope.go:122] "RemoveContainer" containerID="d217dd5d95f2f5519a2b27c5c6188ac4f0e89daf1faade31eee041cda9eae8c5" Oct 30 14:30:17.309514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.309419 5824 scope.go:122] "RemoveContainer" containerID="1a7a55e4a34a39fcda37d15034313008b90b4cc42dd3d7b8f3c0b828d6fb3b25" Oct 30 14:30:17.309648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.309629 5824 scope.go:122] "RemoveContainer" containerID="fc8171c4dce2ec1dbc6eab69fb9fd044a91c0731e5e84c61de5fb7cfe6dfd57f" Oct 30 14:30:17.591827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.591757 5824 generic.go:356] "Generic (PLEG): container finished" podID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerID="83e8cbbea7cf1e17c069adbaa750e7461dcfa1e89628c56ca87d0eba262449d3" exitCode=60 Oct 30 14:30:17.591827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.591809 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" event={"ID":"a8f21dc7-a231-42c9-a073-19ad3fb9f22b","Type":"ContainerDied","Data":"83e8cbbea7cf1e17c069adbaa750e7461dcfa1e89628c56ca87d0eba262449d3"} Oct 30 14:30:17.592351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.591866 5824 scope.go:122] "RemoveContainer" containerID="5abae36e5409c58300f460761d629c761efcdef1c9c7c8ca7d7c5aeab8a16839" Oct 30 14:30:17.592411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.592350 5824 scope.go:122] "RemoveContainer" containerID="83e8cbbea7cf1e17c069adbaa750e7461dcfa1e89628c56ca87d0eba262449d3" Oct 30 14:30:17.593073 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:17.592781 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl_odh-ai-gateway-infra(a8f21dc7-a231-42c9-a073-19ad3fb9f22b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" Oct 30 14:30:17.594273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.594244 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" event={"ID":"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f","Type":"ContainerStarted","Data":"2d9f6e3912889e40222a5707d2e6f6d4f4ba3b836dd13da706bd5d954c994b30"} Oct 30 14:30:17.596451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.596421 5824 generic.go:356] "Generic (PLEG): container finished" podID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerID="beba06e2364ad343441c6f80f14ebcf7ecf37ebd431cb2608826d061d571d2d7" exitCode=60 Oct 30 14:30:17.596554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.596473 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" event={"ID":"b78952e0-14c3-4811-8515-dc14aa6d7449","Type":"ContainerDied","Data":"beba06e2364ad343441c6f80f14ebcf7ecf37ebd431cb2608826d061d571d2d7"} Oct 30 14:30:17.596943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.596923 5824 scope.go:122] "RemoveContainer" containerID="beba06e2364ad343441c6f80f14ebcf7ecf37ebd431cb2608826d061d571d2d7" Oct 30 14:30:17.597482 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:17.597456 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9_odh-ai-gateway-infra(b78952e0-14c3-4811-8515-dc14aa6d7449)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" Oct 30 14:30:17.598832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.598808 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" event={"ID":"9a085f71-14d2-4210-b228-abd9eb70f463","Type":"ContainerStarted","Data":"3616064518b76489cf60a8333266aeddd031ac2d09f4384d6eead41155adc99c"} Oct 30 14:30:17.662960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.662938 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh"] Oct 30 14:30:17.715549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.715518 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k"] Oct 30 14:30:17.870462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.870428 5824 scope.go:122] "RemoveContainer" containerID="d44022c0e54d14f3782d009a8209b519f5be4127984e527f97d2a22370b6850a" Oct 30 14:30:17.919911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.918529 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:17.984345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:17.984310 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:18.020692 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.020655 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1e643a-9a04-4a09-8d90-ccccdd02977c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1e643a-9a04-4a09-8d90-ccccdd02977c-maas-api-service-ca\") pod \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\" (UID: \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\") " Oct 30 14:30:18.020850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.020757 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1e643a-9a04-4a09-8d90-ccccdd02977c-kube-api-access-h7qtf\" (UniqueName: \"kubernetes.io/projected/4a1e643a-9a04-4a09-8d90-ccccdd02977c-kube-api-access-h7qtf\") pod \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\" (UID: \"4a1e643a-9a04-4a09-8d90-ccccdd02977c\") " Oct 30 14:30:18.023400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.023370 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1e643a-9a04-4a09-8d90-ccccdd02977c-maas-api-service-ca" pod "4a1e643a-9a04-4a09-8d90-ccccdd02977c" (UID: "4a1e643a-9a04-4a09-8d90-ccccdd02977c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:18.042778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.042742 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1e643a-9a04-4a09-8d90-ccccdd02977c-kube-api-access-h7qtf" pod "4a1e643a-9a04-4a09-8d90-ccccdd02977c" (UID: "4a1e643a-9a04-4a09-8d90-ccccdd02977c"). InnerVolumeSpecName "kube-api-access-h7qtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:18.123404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.123329 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-kube-api-access-4j7wp\" (UniqueName: \"kubernetes.io/projected/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-kube-api-access-4j7wp\") pod \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\" (UID: \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\") " Oct 30 14:30:18.123533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.123489 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-maas-api-service-ca\") pod \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\" (UID: \"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b\") " Oct 30 14:30:18.124488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.123868 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-maas-api-service-ca" pod "dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" (UID: "dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:18.124488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.124399 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1e643a-9a04-4a09-8d90-ccccdd02977c-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.124488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.124423 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7qtf\" (UniqueName: \"kubernetes.io/projected/4a1e643a-9a04-4a09-8d90-ccccdd02977c-kube-api-access-h7qtf\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.124488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.124438 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.125576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.125553 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-kube-api-access-4j7wp" pod "dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" (UID: "dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b"). InnerVolumeSpecName "kube-api-access-4j7wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:18.225990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.225961 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j7wp\" (UniqueName: \"kubernetes.io/projected/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b-kube-api-access-4j7wp\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:18.309801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.309776 5824 scope.go:122] "RemoveContainer" containerID="19cd0d7e80cf1dd844db19723559cfe4b95b0ed9b1a7e461d470a4116635a0c5" Oct 30 14:30:18.612743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.612715 5824 generic.go:356] "Generic (PLEG): container finished" podID="9a085f71-14d2-4210-b228-abd9eb70f463" containerID="3616064518b76489cf60a8333266aeddd031ac2d09f4384d6eead41155adc99c" exitCode=60 Oct 30 14:30:18.613158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.612790 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" event={"ID":"9a085f71-14d2-4210-b228-abd9eb70f463","Type":"ContainerDied","Data":"3616064518b76489cf60a8333266aeddd031ac2d09f4384d6eead41155adc99c"} Oct 30 14:30:18.613158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.612859 5824 scope.go:122] "RemoveContainer" containerID="fba75e3085df0221f46f6a8dbdc2dbb62f3ac887f8a2742fe3d9f03444ee2dd7" Oct 30 14:30:18.613306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.613274 5824 scope.go:122] "RemoveContainer" containerID="3616064518b76489cf60a8333266aeddd031ac2d09f4384d6eead41155adc99c" Oct 30 14:30:18.613668 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:18.613645 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd_odh-ai-gateway-infra(9a085f71-14d2-4210-b228-abd9eb70f463)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" Oct 30 14:30:18.614547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.614527 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" Oct 30 14:30:18.614547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.614540 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh" event={"ID":"4a1e643a-9a04-4a09-8d90-ccccdd02977c","Type":"ContainerDied","Data":"775ff6c15c65dad1c3566b3297e621aa7534c265830ccced2bab2a6b25492f8e"} Oct 30 14:30:18.622104 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.622082 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" event={"ID":"b7a225fb-a633-426a-8a53-f3214aa77efb","Type":"ContainerStarted","Data":"a8e9aac6f27515e59a7852b9a8b6d76f82935795a8626ab0e3426ab1a4f0ce43"} Oct 30 14:30:18.623815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.623792 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" event={"ID":"dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b","Type":"ContainerDied","Data":"b4cd87cb724c3bfac0544ac73c6da82e98000ed13d93a7e1787d6d46c666d22d"} Oct 30 14:30:18.623935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.623799 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k" Oct 30 14:30:18.626043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.626017 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" event={"ID":"5baa41cf-dc7b-4386-a579-92f33e4ae890","Type":"ContainerStarted","Data":"2805b0832a0ffda987aba995f00a941e445e5835a4f57e72cea7c91d91c5af99"} Oct 30 14:30:18.630105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.630081 5824 generic.go:356] "Generic (PLEG): container finished" podID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerID="2d9f6e3912889e40222a5707d2e6f6d4f4ba3b836dd13da706bd5d954c994b30" exitCode=60 Oct 30 14:30:18.630257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.630155 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" event={"ID":"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f","Type":"ContainerDied","Data":"2d9f6e3912889e40222a5707d2e6f6d4f4ba3b836dd13da706bd5d954c994b30"} Oct 30 14:30:18.630614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.630581 5824 scope.go:122] "RemoveContainer" containerID="2d9f6e3912889e40222a5707d2e6f6d4f4ba3b836dd13da706bd5d954c994b30" Oct 30 14:30:18.630886 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:18.630849 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl_odh-ai-gateway-infra(cadd310c-c30e-4ad2-b5b3-2af8623e7e3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" Oct 30 14:30:18.631062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.631037 5824 scope.go:122] "RemoveContainer" containerID="26949573c451e056822082059879023086383abb5b7e282e38470d791e9f0025" Oct 30 14:30:18.634848 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.634801 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" event={"ID":"ca50db98-1579-42c1-8e66-1ebb6e7c5323","Type":"ContainerStarted","Data":"60f7db65c131301e8fab7296c7f38d245efb6de08a1634507cf2f662355e1e20"} Oct 30 14:30:18.643571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.643555 5824 scope.go:122] "RemoveContainer" containerID="39ba5132852b71a44353386d19707154485e2a45a4ecd63a110851bc87cd7fb2" Oct 30 14:30:18.681984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.681962 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl"] Oct 30 14:30:18.706238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.706209 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh"] Oct 30 14:30:18.717168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.717138 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-0b1c15-29889510-t5whh"] Oct 30 14:30:18.725673 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.725652 5824 scope.go:122] "RemoveContainer" containerID="8a5c2db06132e5f1fe458457f83930ee3612deed7c8c5b117fde46b6f3fb3918" Oct 30 14:30:18.804671 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.804645 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl"] Oct 30 14:30:18.858495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.858470 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9"] Oct 30 14:30:18.869875 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.869851 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k"] Oct 30 14:30:18.876307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.876281 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e371c3-29889510-trw5k"] Oct 30 14:30:18.905877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.905852 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd"] Oct 30 14:30:18.906449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:18.906426 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:19.043548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.043166 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-kube-api-access-frk2v\" (UniqueName: \"kubernetes.io/projected/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-kube-api-access-frk2v\") pod \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\" (UID: \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\") " Oct 30 14:30:19.043548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.043484 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-maas-api-service-ca\") pod \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\" (UID: \"a8f21dc7-a231-42c9-a073-19ad3fb9f22b\") " Oct 30 14:30:19.044676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.043977 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-maas-api-service-ca" pod "a8f21dc7-a231-42c9-a073-19ad3fb9f22b" (UID: "a8f21dc7-a231-42c9-a073-19ad3fb9f22b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:19.044676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.044156 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:19.045611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.045580 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-kube-api-access-frk2v" pod "a8f21dc7-a231-42c9-a073-19ad3fb9f22b" (UID: "a8f21dc7-a231-42c9-a073-19ad3fb9f22b"). InnerVolumeSpecName "kube-api-access-frk2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:19.110969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.110942 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:19.145912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.145851 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frk2v\" (UniqueName: \"kubernetes.io/projected/a8f21dc7-a231-42c9-a073-19ad3fb9f22b-kube-api-access-frk2v\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:19.246646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.246619 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78952e0-14c3-4811-8515-dc14aa6d7449-kube-api-access-qfb9l\" (UniqueName: \"kubernetes.io/projected/b78952e0-14c3-4811-8515-dc14aa6d7449-kube-api-access-qfb9l\") pod \"b78952e0-14c3-4811-8515-dc14aa6d7449\" (UID: \"b78952e0-14c3-4811-8515-dc14aa6d7449\") " Oct 30 14:30:19.246813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.246792 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78952e0-14c3-4811-8515-dc14aa6d7449-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78952e0-14c3-4811-8515-dc14aa6d7449-maas-api-service-ca\") pod \"b78952e0-14c3-4811-8515-dc14aa6d7449\" (UID: \"b78952e0-14c3-4811-8515-dc14aa6d7449\") " Oct 30 14:30:19.247164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.247137 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78952e0-14c3-4811-8515-dc14aa6d7449-maas-api-service-ca" pod "b78952e0-14c3-4811-8515-dc14aa6d7449" (UID: "b78952e0-14c3-4811-8515-dc14aa6d7449"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:19.247565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.247544 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78952e0-14c3-4811-8515-dc14aa6d7449-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:19.248785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.248760 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78952e0-14c3-4811-8515-dc14aa6d7449-kube-api-access-qfb9l" pod "b78952e0-14c3-4811-8515-dc14aa6d7449" (UID: "b78952e0-14c3-4811-8515-dc14aa6d7449"). InnerVolumeSpecName "kube-api-access-qfb9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:19.309928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.309902 5824 scope.go:122] "RemoveContainer" containerID="d9f40356054d0e44d6d7f55a7609490f07bf42e977ddf5c86c6810c17799cf08" Oct 30 14:30:19.311482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.310834 5824 scope.go:122] "RemoveContainer" containerID="5eef11ab9b3b88c0ff5645bd7e12f12c899a1a4dada49eaa62dd71696451c02a" Oct 30 14:30:19.349382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.349358 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfb9l\" (UniqueName: \"kubernetes.io/projected/b78952e0-14c3-4811-8515-dc14aa6d7449-kube-api-access-qfb9l\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:19.645625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.645566 5824 generic.go:356] "Generic (PLEG): container finished" podID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerID="8af22f062d0c5c139555074f8a148a4b0ce0f43b4ed03e6d90af2900e5266a33" exitCode=60 Oct 30 14:30:19.645625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.645606 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" event={"ID":"5ab9d46c-29a1-479a-a477-ae3f53404b91","Type":"ContainerDied","Data":"8af22f062d0c5c139555074f8a148a4b0ce0f43b4ed03e6d90af2900e5266a33"} Oct 30 14:30:19.646117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.645672 5824 scope.go:122] "RemoveContainer" containerID="19cd0d7e80cf1dd844db19723559cfe4b95b0ed9b1a7e461d470a4116635a0c5" Oct 30 14:30:19.646164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.646143 5824 scope.go:122] "RemoveContainer" containerID="8af22f062d0c5c139555074f8a148a4b0ce0f43b4ed03e6d90af2900e5266a33" Oct 30 14:30:19.646569 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:19.646544 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz_odh-ai-gateway-infra(5ab9d46c-29a1-479a-a477-ae3f53404b91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" Oct 30 14:30:19.651422 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.651399 5824 generic.go:356] "Generic (PLEG): container finished" podID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerID="a8e9aac6f27515e59a7852b9a8b6d76f82935795a8626ab0e3426ab1a4f0ce43" exitCode=60 Oct 30 14:30:19.651531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.651474 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" event={"ID":"b7a225fb-a633-426a-8a53-f3214aa77efb","Type":"ContainerDied","Data":"a8e9aac6f27515e59a7852b9a8b6d76f82935795a8626ab0e3426ab1a4f0ce43"} Oct 30 14:30:19.651921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.651902 5824 scope.go:122] "RemoveContainer" containerID="a8e9aac6f27515e59a7852b9a8b6d76f82935795a8626ab0e3426ab1a4f0ce43" Oct 30 14:30:19.652302 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:19.652279 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl_odh-ai-gateway-infra(b7a225fb-a633-426a-8a53-f3214aa77efb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" Oct 30 14:30:19.655266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.655242 5824 generic.go:356] "Generic (PLEG): container finished" podID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerID="2805b0832a0ffda987aba995f00a941e445e5835a4f57e72cea7c91d91c5af99" exitCode=60 Oct 30 14:30:19.655360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.655309 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" event={"ID":"5baa41cf-dc7b-4386-a579-92f33e4ae890","Type":"ContainerDied","Data":"2805b0832a0ffda987aba995f00a941e445e5835a4f57e72cea7c91d91c5af99"} Oct 30 14:30:19.656150 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.656127 5824 scope.go:122] "RemoveContainer" containerID="2805b0832a0ffda987aba995f00a941e445e5835a4f57e72cea7c91d91c5af99" Oct 30 14:30:19.656413 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:19.656387 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw_odh-ai-gateway-infra(5baa41cf-dc7b-4386-a579-92f33e4ae890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" Oct 30 14:30:19.657005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.656983 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" event={"ID":"a8f21dc7-a231-42c9-a073-19ad3fb9f22b","Type":"ContainerDied","Data":"54d7ee5a41261408bc38629c8d1975b4ee83863a2acff8e2827bfc8b9d5f88f3"} Oct 30 14:30:19.657107 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.657027 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl" Oct 30 14:30:19.668179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.662557 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" event={"ID":"b78952e0-14c3-4811-8515-dc14aa6d7449","Type":"ContainerDied","Data":"11b05156c21a28e847340db7964f63d51cb31104787ba34dd536745d2b00acdf"} Oct 30 14:30:19.668179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.662543 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9" Oct 30 14:30:19.671429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.671405 5824 generic.go:356] "Generic (PLEG): container finished" podID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerID="60f7db65c131301e8fab7296c7f38d245efb6de08a1634507cf2f662355e1e20" exitCode=60 Oct 30 14:30:19.671525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.671466 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" event={"ID":"ca50db98-1579-42c1-8e66-1ebb6e7c5323","Type":"ContainerDied","Data":"60f7db65c131301e8fab7296c7f38d245efb6de08a1634507cf2f662355e1e20"} Oct 30 14:30:19.671855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.671830 5824 scope.go:122] "RemoveContainer" containerID="60f7db65c131301e8fab7296c7f38d245efb6de08a1634507cf2f662355e1e20" Oct 30 14:30:19.672139 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:19.672101 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq_odh-ai-gateway-infra(ca50db98-1579-42c1-8e66-1ebb6e7c5323)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" Oct 30 14:30:19.673815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.673791 5824 scope.go:122] "RemoveContainer" containerID="d217dd5d95f2f5519a2b27c5c6188ac4f0e89daf1faade31eee041cda9eae8c5" Oct 30 14:30:19.706497 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.706475 5824 scope.go:122] "RemoveContainer" containerID="1a7a55e4a34a39fcda37d15034313008b90b4cc42dd3d7b8f3c0b828d6fb3b25" Oct 30 14:30:19.728904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.728844 5824 scope.go:122] "RemoveContainer" containerID="83e8cbbea7cf1e17c069adbaa750e7461dcfa1e89628c56ca87d0eba262449d3" Oct 30 14:30:19.763437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.761015 5824 scope.go:122] "RemoveContainer" containerID="beba06e2364ad343441c6f80f14ebcf7ecf37ebd431cb2608826d061d571d2d7" Oct 30 14:30:19.774546 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.774519 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq"] Oct 30 14:30:19.785659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.785638 5824 scope.go:122] "RemoveContainer" containerID="fc8171c4dce2ec1dbc6eab69fb9fd044a91c0731e5e84c61de5fb7cfe6dfd57f" Oct 30 14:30:19.789156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.789130 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl"] Oct 30 14:30:19.807287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.806861 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889510-g9ztl"] Oct 30 14:30:19.882180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.882136 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:19.910644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.910624 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:19.932315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.932289 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw"] Oct 30 14:30:19.936721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.936697 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9"] Oct 30 14:30:19.941392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.941369 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ca229c-29889510-5mcm9"] Oct 30 14:30:19.958160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.958091 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-kube-api-access-ghpd4\" (UniqueName: \"kubernetes.io/projected/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-kube-api-access-ghpd4\") pod \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\" (UID: \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\") " Oct 30 14:30:19.958293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.958206 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a085f71-14d2-4210-b228-abd9eb70f463-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a085f71-14d2-4210-b228-abd9eb70f463-maas-api-service-ca\") pod \"9a085f71-14d2-4210-b228-abd9eb70f463\" (UID: \"9a085f71-14d2-4210-b228-abd9eb70f463\") " Oct 30 14:30:19.958390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.958372 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a085f71-14d2-4210-b228-abd9eb70f463-kube-api-access-xqlvm\" (UniqueName: \"kubernetes.io/projected/9a085f71-14d2-4210-b228-abd9eb70f463-kube-api-access-xqlvm\") pod \"9a085f71-14d2-4210-b228-abd9eb70f463\" (UID: \"9a085f71-14d2-4210-b228-abd9eb70f463\") " Oct 30 14:30:19.958495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.958480 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-maas-api-service-ca\") pod \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\" (UID: \"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f\") " Oct 30 14:30:19.958557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.958513 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a085f71-14d2-4210-b228-abd9eb70f463-maas-api-service-ca" pod "9a085f71-14d2-4210-b228-abd9eb70f463" (UID: "9a085f71-14d2-4210-b228-abd9eb70f463"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:19.958906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.958879 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-maas-api-service-ca" pod "cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" (UID: "cadd310c-c30e-4ad2-b5b3-2af8623e7e3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:19.959166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.959142 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a085f71-14d2-4210-b228-abd9eb70f463-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:19.959273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.959173 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:19.960360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.960339 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-kube-api-access-ghpd4" pod "cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" (UID: "cadd310c-c30e-4ad2-b5b3-2af8623e7e3f"). InnerVolumeSpecName "kube-api-access-ghpd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:19.960614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.960595 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a085f71-14d2-4210-b228-abd9eb70f463-kube-api-access-xqlvm" pod "9a085f71-14d2-4210-b228-abd9eb70f463" (UID: "9a085f71-14d2-4210-b228-abd9eb70f463"). InnerVolumeSpecName "kube-api-access-xqlvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:19.981399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:19.981378 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl"] Oct 30 14:30:20.060371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.060343 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqlvm\" (UniqueName: \"kubernetes.io/projected/9a085f71-14d2-4210-b228-abd9eb70f463-kube-api-access-xqlvm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:20.060371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.060369 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghpd4\" (UniqueName: \"kubernetes.io/projected/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f-kube-api-access-ghpd4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:20.320716 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.320635 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" path="/var/lib/kubelet/pods/4a1e643a-9a04-4a09-8d90-ccccdd02977c/volumes" Oct 30 14:30:20.321155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.321131 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" path="/var/lib/kubelet/pods/a8f21dc7-a231-42c9-a073-19ad3fb9f22b/volumes" Oct 30 14:30:20.321617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.321595 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" path="/var/lib/kubelet/pods/b78952e0-14c3-4811-8515-dc14aa6d7449/volumes" Oct 30 14:30:20.322365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.322343 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" path="/var/lib/kubelet/pods/dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b/volumes" Oct 30 14:30:20.682850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.682824 5824 generic.go:356] "Generic (PLEG): container finished" podID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerID="f3e9a33276ac309763a1efea106178a5f74455a21ba350d7f74ac7094b4a9d66" exitCode=60 Oct 30 14:30:20.683295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.682893 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" event={"ID":"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d","Type":"ContainerDied","Data":"f3e9a33276ac309763a1efea106178a5f74455a21ba350d7f74ac7094b4a9d66"} Oct 30 14:30:20.683295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.682953 5824 scope.go:122] "RemoveContainer" containerID="5eef11ab9b3b88c0ff5645bd7e12f12c899a1a4dada49eaa62dd71696451c02a" Oct 30 14:30:20.683420 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.683330 5824 scope.go:122] "RemoveContainer" containerID="f3e9a33276ac309763a1efea106178a5f74455a21ba350d7f74ac7094b4a9d66" Oct 30 14:30:20.683598 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:20.683575 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd_odh-ai-gateway-infra(ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" Oct 30 14:30:20.687540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.687519 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" Oct 30 14:30:20.687634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.687540 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl" event={"ID":"cadd310c-c30e-4ad2-b5b3-2af8623e7e3f","Type":"ContainerDied","Data":"175e9f9909fd2dbf44d2002ca42a99f01c4853c3758a3dc94e84d97dcd512d94"} Oct 30 14:30:20.694250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.694223 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" event={"ID":"9a085f71-14d2-4210-b228-abd9eb70f463","Type":"ContainerDied","Data":"7e9f4ca090cd8f93b2f0ab3d3102b0710e31fe0c471030fbb329ca40a711fb13"} Oct 30 14:30:20.694372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.694259 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd" Oct 30 14:30:20.698178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.698147 5824 generic.go:356] "Generic (PLEG): container finished" podID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerID="318a084f950e383a756c817e8891e42e138d16403d322fc2ddfd442e16ec5960" exitCode=60 Oct 30 14:30:20.698324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.698267 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" event={"ID":"59bf5cee-4198-421d-9b9f-633961ddba6d","Type":"ContainerDied","Data":"318a084f950e383a756c817e8891e42e138d16403d322fc2ddfd442e16ec5960"} Oct 30 14:30:20.699261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.698603 5824 scope.go:122] "RemoveContainer" containerID="318a084f950e383a756c817e8891e42e138d16403d322fc2ddfd442e16ec5960" Oct 30 14:30:20.699261 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:20.698840 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs_odh-ai-gateway-infra(59bf5cee-4198-421d-9b9f-633961ddba6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" Oct 30 14:30:20.702699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.702677 5824 scope.go:122] "RemoveContainer" containerID="2d9f6e3912889e40222a5707d2e6f6d4f4ba3b836dd13da706bd5d954c994b30" Oct 30 14:30:20.710867 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.710843 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz"] Oct 30 14:30:20.718260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.718047 5824 scope.go:122] "RemoveContainer" containerID="3616064518b76489cf60a8333266aeddd031ac2d09f4384d6eead41155adc99c" Oct 30 14:30:20.737624 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.737230 5824 scope.go:122] "RemoveContainer" containerID="d9f40356054d0e44d6d7f55a7609490f07bf42e977ddf5c86c6810c17799cf08" Oct 30 14:30:20.977834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:20.977804 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:21.006367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.006337 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd"] Oct 30 14:30:21.016058 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.016027 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889510-84bdd"] Oct 30 14:30:21.066091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.066065 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:21.074050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.074021 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:21.079043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.079016 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:21.099545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.099519 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl"] Oct 30 14:30:21.106290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.106266 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889510-9p7cl"] Oct 30 14:30:21.106808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.106785 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca50db98-1579-42c1-8e66-1ebb6e7c5323-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca50db98-1579-42c1-8e66-1ebb6e7c5323-maas-api-service-ca\") pod \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\" (UID: \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\") " Oct 30 14:30:21.106905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.106892 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5baa41cf-dc7b-4386-a579-92f33e4ae890-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baa41cf-dc7b-4386-a579-92f33e4ae890-maas-api-service-ca\") pod \"5baa41cf-dc7b-4386-a579-92f33e4ae890\" (UID: \"5baa41cf-dc7b-4386-a579-92f33e4ae890\") " Oct 30 14:30:21.106980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.106951 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a225fb-a633-426a-8a53-f3214aa77efb-kube-api-access-rpxgz\" (UniqueName: \"kubernetes.io/projected/b7a225fb-a633-426a-8a53-f3214aa77efb-kube-api-access-rpxgz\") pod \"b7a225fb-a633-426a-8a53-f3214aa77efb\" (UID: \"b7a225fb-a633-426a-8a53-f3214aa77efb\") " Oct 30 14:30:21.107039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.106982 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca50db98-1579-42c1-8e66-1ebb6e7c5323-kube-api-access-mp55t\" (UniqueName: \"kubernetes.io/projected/ca50db98-1579-42c1-8e66-1ebb6e7c5323-kube-api-access-mp55t\") pod \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\" (UID: \"ca50db98-1579-42c1-8e66-1ebb6e7c5323\") " Oct 30 14:30:21.107039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.107013 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5baa41cf-dc7b-4386-a579-92f33e4ae890-kube-api-access-n8c9x\" (UniqueName: \"kubernetes.io/projected/5baa41cf-dc7b-4386-a579-92f33e4ae890-kube-api-access-n8c9x\") pod \"5baa41cf-dc7b-4386-a579-92f33e4ae890\" (UID: \"5baa41cf-dc7b-4386-a579-92f33e4ae890\") " Oct 30 14:30:21.107168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.107036 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a225fb-a633-426a-8a53-f3214aa77efb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a225fb-a633-426a-8a53-f3214aa77efb-maas-api-service-ca\") pod \"b7a225fb-a633-426a-8a53-f3214aa77efb\" (UID: \"b7a225fb-a633-426a-8a53-f3214aa77efb\") " Oct 30 14:30:21.107168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.107151 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab9d46c-29a1-479a-a477-ae3f53404b91-kube-api-access-lqsdk\" (UniqueName: \"kubernetes.io/projected/5ab9d46c-29a1-479a-a477-ae3f53404b91-kube-api-access-lqsdk\") pod \"5ab9d46c-29a1-479a-a477-ae3f53404b91\" (UID: \"5ab9d46c-29a1-479a-a477-ae3f53404b91\") " Oct 30 14:30:21.107284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.107185 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab9d46c-29a1-479a-a477-ae3f53404b91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9d46c-29a1-479a-a477-ae3f53404b91-maas-api-service-ca\") pod \"5ab9d46c-29a1-479a-a477-ae3f53404b91\" (UID: \"5ab9d46c-29a1-479a-a477-ae3f53404b91\") " Oct 30 14:30:21.107284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.107213 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca50db98-1579-42c1-8e66-1ebb6e7c5323-maas-api-service-ca" pod "ca50db98-1579-42c1-8e66-1ebb6e7c5323" (UID: "ca50db98-1579-42c1-8e66-1ebb6e7c5323"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:21.107415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.107300 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baa41cf-dc7b-4386-a579-92f33e4ae890-maas-api-service-ca" pod "5baa41cf-dc7b-4386-a579-92f33e4ae890" (UID: "5baa41cf-dc7b-4386-a579-92f33e4ae890"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:21.107614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.107580 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a225fb-a633-426a-8a53-f3214aa77efb-maas-api-service-ca" pod "b7a225fb-a633-426a-8a53-f3214aa77efb" (UID: "b7a225fb-a633-426a-8a53-f3214aa77efb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:21.107934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.107915 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab9d46c-29a1-479a-a477-ae3f53404b91-maas-api-service-ca" pod "5ab9d46c-29a1-479a-a477-ae3f53404b91" (UID: "5ab9d46c-29a1-479a-a477-ae3f53404b91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:21.108398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.108372 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a225fb-a633-426a-8a53-f3214aa77efb-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.108496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.108401 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab9d46c-29a1-479a-a477-ae3f53404b91-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.108496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.108417 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca50db98-1579-42c1-8e66-1ebb6e7c5323-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.108496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.108431 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5baa41cf-dc7b-4386-a579-92f33e4ae890-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.109789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.109763 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca50db98-1579-42c1-8e66-1ebb6e7c5323-kube-api-access-mp55t" pod "ca50db98-1579-42c1-8e66-1ebb6e7c5323" (UID: "ca50db98-1579-42c1-8e66-1ebb6e7c5323"). InnerVolumeSpecName "kube-api-access-mp55t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:21.109889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.109805 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baa41cf-dc7b-4386-a579-92f33e4ae890-kube-api-access-n8c9x" pod "5baa41cf-dc7b-4386-a579-92f33e4ae890" (UID: "5baa41cf-dc7b-4386-a579-92f33e4ae890"). InnerVolumeSpecName "kube-api-access-n8c9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:21.110805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.110781 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab9d46c-29a1-479a-a477-ae3f53404b91-kube-api-access-lqsdk" pod "5ab9d46c-29a1-479a-a477-ae3f53404b91" (UID: "5ab9d46c-29a1-479a-a477-ae3f53404b91"). InnerVolumeSpecName "kube-api-access-lqsdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:21.119370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.119336 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a225fb-a633-426a-8a53-f3214aa77efb-kube-api-access-rpxgz" pod "b7a225fb-a633-426a-8a53-f3214aa77efb" (UID: "b7a225fb-a633-426a-8a53-f3214aa77efb"). InnerVolumeSpecName "kube-api-access-rpxgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:21.209508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.209419 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxgz\" (UniqueName: \"kubernetes.io/projected/b7a225fb-a633-426a-8a53-f3214aa77efb-kube-api-access-rpxgz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.209508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.209447 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp55t\" (UniqueName: \"kubernetes.io/projected/ca50db98-1579-42c1-8e66-1ebb6e7c5323-kube-api-access-mp55t\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.209508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.209457 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8c9x\" (UniqueName: \"kubernetes.io/projected/5baa41cf-dc7b-4386-a579-92f33e4ae890-kube-api-access-n8c9x\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.209508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.209466 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqsdk\" (UniqueName: \"kubernetes.io/projected/5ab9d46c-29a1-479a-a477-ae3f53404b91-kube-api-access-lqsdk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:21.716200 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.716146 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" Oct 30 14:30:21.716679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.716181 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw" event={"ID":"5baa41cf-dc7b-4386-a579-92f33e4ae890","Type":"ContainerDied","Data":"a24209a67a93ded0b2d6ee3d65ee3bf1a7d5c09820f76304a60a68594ee664b9"} Oct 30 14:30:21.716679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.716293 5824 scope.go:122] "RemoveContainer" containerID="2805b0832a0ffda987aba995f00a941e445e5835a4f57e72cea7c91d91c5af99" Oct 30 14:30:21.720813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.720788 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" Oct 30 14:30:21.720958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.720812 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq" event={"ID":"ca50db98-1579-42c1-8e66-1ebb6e7c5323","Type":"ContainerDied","Data":"57bec324adaeae251bb8992ebffb12f27ce4f282ed50a8b0ee359ef74df461c3"} Oct 30 14:30:21.722347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.722322 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" Oct 30 14:30:21.722456 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.722322 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz" event={"ID":"5ab9d46c-29a1-479a-a477-ae3f53404b91","Type":"ContainerDied","Data":"5bae4f68a404796d7222df94def03eb329d34e2ff07a54f083a2d2d684a6041d"} Oct 30 14:30:21.726622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.726597 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" event={"ID":"b7a225fb-a633-426a-8a53-f3214aa77efb","Type":"ContainerDied","Data":"74d6171c2bf993e2574ed272d3832e553ab603c02047f722959b0deb25d2ced8"} Oct 30 14:30:21.726735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.726693 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl" Oct 30 14:30:21.733285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.733172 5824 scope.go:122] "RemoveContainer" containerID="60f7db65c131301e8fab7296c7f38d245efb6de08a1634507cf2f662355e1e20" Oct 30 14:30:21.742013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.741991 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd"] Oct 30 14:30:21.745998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.745981 5824 scope.go:122] "RemoveContainer" containerID="8af22f062d0c5c139555074f8a148a4b0ce0f43b4ed03e6d90af2900e5266a33" Oct 30 14:30:21.756312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.756175 5824 scope.go:122] "RemoveContainer" containerID="a8e9aac6f27515e59a7852b9a8b6d76f82935795a8626ab0e3426ab1a4f0ce43" Oct 30 14:30:21.784023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.783999 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq"] Oct 30 14:30:21.789674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.789650 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-409364-29889510-zvsfq"] Oct 30 14:30:21.832423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.832393 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz"] Oct 30 14:30:21.843578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.843552 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-019e5b-29889510-rgpgz"] Oct 30 14:30:21.880416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.880389 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs"] Oct 30 14:30:21.885320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.885291 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw"] Oct 30 14:30:21.892025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.891999 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889510-xgkhw"] Oct 30 14:30:21.935984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:21.935959 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:22.019823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.019756 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-maas-api-service-ca\") pod \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\" (UID: \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\") " Oct 30 14:30:22.020005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.019988 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-kube-api-access-mxc7f\" (UniqueName: \"kubernetes.io/projected/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-kube-api-access-mxc7f\") pod \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\" (UID: \"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d\") " Oct 30 14:30:22.020172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.020147 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-maas-api-service-ca" pod "ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" (UID: "ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:22.020669 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.020616 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.022578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.022552 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-kube-api-access-mxc7f" pod "ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" (UID: "ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d"). InnerVolumeSpecName "kube-api-access-mxc7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:22.064243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.064218 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:22.066643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.066620 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl"] Oct 30 14:30:22.100212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.083588 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5704d6-29889510-kqzsl"] Oct 30 14:30:22.121787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.121763 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bf5cee-4198-421d-9b9f-633961ddba6d-kube-api-access-8h7v7\" (UniqueName: \"kubernetes.io/projected/59bf5cee-4198-421d-9b9f-633961ddba6d-kube-api-access-8h7v7\") pod \"59bf5cee-4198-421d-9b9f-633961ddba6d\" (UID: \"59bf5cee-4198-421d-9b9f-633961ddba6d\") " Oct 30 14:30:22.121948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.121823 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bf5cee-4198-421d-9b9f-633961ddba6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bf5cee-4198-421d-9b9f-633961ddba6d-maas-api-service-ca\") pod \"59bf5cee-4198-421d-9b9f-633961ddba6d\" (UID: \"59bf5cee-4198-421d-9b9f-633961ddba6d\") " Oct 30 14:30:22.122150 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.122128 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bf5cee-4198-421d-9b9f-633961ddba6d-maas-api-service-ca" pod "59bf5cee-4198-421d-9b9f-633961ddba6d" (UID: "59bf5cee-4198-421d-9b9f-633961ddba6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:22.122633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.122611 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxc7f\" (UniqueName: \"kubernetes.io/projected/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d-kube-api-access-mxc7f\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.122736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.122635 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bf5cee-4198-421d-9b9f-633961ddba6d-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.123957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.123936 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bf5cee-4198-421d-9b9f-633961ddba6d-kube-api-access-8h7v7" pod "59bf5cee-4198-421d-9b9f-633961ddba6d" (UID: "59bf5cee-4198-421d-9b9f-633961ddba6d"). InnerVolumeSpecName "kube-api-access-8h7v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:22.223881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.223849 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7v7\" (UniqueName: \"kubernetes.io/projected/59bf5cee-4198-421d-9b9f-633961ddba6d-kube-api-access-8h7v7\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:22.318924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.318847 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" path="/var/lib/kubelet/pods/5ab9d46c-29a1-479a-a477-ae3f53404b91/volumes" Oct 30 14:30:22.319444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.319427 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" path="/var/lib/kubelet/pods/5baa41cf-dc7b-4386-a579-92f33e4ae890/volumes" Oct 30 14:30:22.319886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.319871 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" path="/var/lib/kubelet/pods/9a085f71-14d2-4210-b228-abd9eb70f463/volumes" Oct 30 14:30:22.320579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.320563 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" path="/var/lib/kubelet/pods/b7a225fb-a633-426a-8a53-f3214aa77efb/volumes" Oct 30 14:30:22.320984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.320971 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" path="/var/lib/kubelet/pods/ca50db98-1579-42c1-8e66-1ebb6e7c5323/volumes" Oct 30 14:30:22.321395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.321381 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" path="/var/lib/kubelet/pods/cadd310c-c30e-4ad2-b5b3-2af8623e7e3f/volumes" Oct 30 14:30:22.743674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.743639 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" event={"ID":"59bf5cee-4198-421d-9b9f-633961ddba6d","Type":"ContainerDied","Data":"763858af639183ab838c2f7b583eb831dc8ed3ef8ab597f8db54ee64f048290b"} Oct 30 14:30:22.743674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.743662 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs" Oct 30 14:30:22.744169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.743698 5824 scope.go:122] "RemoveContainer" containerID="318a084f950e383a756c817e8891e42e138d16403d322fc2ddfd442e16ec5960" Oct 30 14:30:22.746442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.746419 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" Oct 30 14:30:22.746569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.746469 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd" event={"ID":"ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d","Type":"ContainerDied","Data":"76863803254c816a16968ded1fdecedc846adeaa36e3134866369e9d0c808e41"} Oct 30 14:30:22.757028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.757011 5824 scope.go:122] "RemoveContainer" containerID="f3e9a33276ac309763a1efea106178a5f74455a21ba350d7f74ac7094b4a9d66" Oct 30 14:30:22.806174 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.806143 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs"] Oct 30 14:30:22.809610 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.809587 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00926c-29889510-5sgbs"] Oct 30 14:30:22.845561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.845530 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd"] Oct 30 14:30:22.853028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:22.852995 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3c580-29889510-mvcdd"] Oct 30 14:30:23.075475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.075409 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq"] Oct 30 14:30:23.076153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076140 5824 cpu_manager.go:414] "containerMap: removing container" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.076224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076154 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.076224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076170 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:23.076224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076176 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:23.076224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076199 5824 cpu_manager.go:414] "containerMap: removing container" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:23.076224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076205 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:23.076224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076215 5824 cpu_manager.go:414] "containerMap: removing container" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:23.076224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076220 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076229 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076234 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076243 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076247 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076257 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076262 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076267 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076272 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076279 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076284 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076288 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076293 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076298 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076304 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076309 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076313 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076325 5824 cpu_manager.go:414] "containerMap: removing container" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076329 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076338 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076342 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076349 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076354 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076360 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076365 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076373 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076378 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076383 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076387 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076396 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076400 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076408 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076413 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076419 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076423 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076430 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076435 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076445 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076449 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:23.076434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076455 5824 cpu_manager.go:414] "containerMap: removing container" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076460 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076474 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076478 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076489 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076493 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076505 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076510 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076518 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076522 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076629 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076635 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076642 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076647 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076654 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076659 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076664 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076669 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076677 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076683 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076689 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076694 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076701 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076706 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076712 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076718 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076723 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076727 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076732 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076738 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076744 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076749 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076754 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076760 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076765 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076771 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076775 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076781 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca50db98-1579-42c1-8e66-1ebb6e7c5323" containerName="cleanup" Oct 30 14:30:23.077535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.076788 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadd310c-c30e-4ad2-b5b3-2af8623e7e3f" containerName="cleanup" Oct 30 14:30:23.081973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.081958 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.099531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.099506 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fde75d-openshift-default-dockercfg-dp8mb\"" Oct 30 14:30:23.103400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.103376 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq"] Oct 30 14:30:23.134425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.134397 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-data\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.134565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.134510 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-token\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.134641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.134600 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-envoy\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.134703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.134678 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-podinfo\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.134811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.134791 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-credential-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.134852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.134839 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.134901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.134870 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istiod-ca-cert\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.135024 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.135006 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlk76\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-kube-api-access-zlk76\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.135156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.135140 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-certs\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.236697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236670 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlk76\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-kube-api-access-zlk76\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.236865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236743 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-certs\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.236865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236785 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-data\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.236865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236819 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-token\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.236865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236862 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-envoy\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236883 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-podinfo\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236936 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-credential-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236974 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.236996 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istiod-ca-cert\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.237246 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-data\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.237290 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-certs\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.237385 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.237518 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-credential-socket\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.237783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.237762 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istiod-ca-cert\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.239154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.239133 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-envoy\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.239439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.239417 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-podinfo\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.262701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.262676 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-token\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.263381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.263359 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlk76\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-kube-api-access-zlk76\") pod \"e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.399812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.399755 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:23.734334 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:30:23.734301 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159f5cf3_e422_4f1a_9769_cb20b007d4b0.slice/crio-04a0fb5518e0a27f6d3ea297c1b2047501660a7ba52cc450327b7294636c0b96 WatchSource:0}: Error finding container 04a0fb5518e0a27f6d3ea297c1b2047501660a7ba52cc450327b7294636c0b96: Status 404 returned error can't find the container with id 04a0fb5518e0a27f6d3ea297c1b2047501660a7ba52cc450327b7294636c0b96 Oct 30 14:30:23.734995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.734966 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq"] Oct 30 14:30:23.737625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.737593 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:30:23.737738 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.737656 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:30:23.737738 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.737687 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:30:23.758612 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:23.758588 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" event={"ID":"159f5cf3-e422-4f1a-9769-cb20b007d4b0","Type":"ContainerStarted","Data":"04a0fb5518e0a27f6d3ea297c1b2047501660a7ba52cc450327b7294636c0b96"} Oct 30 14:30:24.321235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:24.321178 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" path="/var/lib/kubelet/pods/59bf5cee-4198-421d-9b9f-633961ddba6d/volumes" Oct 30 14:30:24.321772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:24.321748 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d" path="/var/lib/kubelet/pods/ef1d7d2a-14d6-44e0-8c24-81216fcc6a1d/volumes" Oct 30 14:30:24.832622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:24.832574 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" event={"ID":"159f5cf3-e422-4f1a-9769-cb20b007d4b0","Type":"ContainerStarted","Data":"2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02"} Oct 30 14:30:24.921244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:24.921152 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" podStartSLOduration=1.921134947 podStartE2EDuration="1.921134947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:24.907747128 +0000 UTC m=+2454.708252609" watchObservedRunningTime="2026-10-30 14:30:24.921134947 +0000 UTC m=+2454.721640394" Oct 30 14:30:25.400930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:25.400898 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:25.402376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:25.402348 5824 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 14:30:25.402492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:25.402406 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 14:30:26.400744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:26.400709 5824 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 14:30:26.401141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:26.400770 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 14:30:27.400465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:27.400422 5824 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 14:30:27.400626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:27.400507 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 14:30:28.400723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:28.400690 5824 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 14:30:28.401086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:28.400744 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 14:30:29.400274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:29.400238 5824 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 14:30:29.400446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:29.400291 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 14:30:30.469299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:30.469257 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:30.469732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:30.469699 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:30.470671 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:30.470649 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:30:33.642352 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:30:33.642312 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:30:52.801203 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:52.801110 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr"] Oct 30 14:30:55.965280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.965247 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks"] Oct 30 14:30:55.966020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966004 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:55.966067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966021 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:55.966067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966045 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:55.966067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966050 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:55.966067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966063 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:55.966067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966069 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966085 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966090 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966112 5824 cpu_manager.go:414] "containerMap: removing container" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966117 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bf5cee-4198-421d-9b9f-633961ddba6d" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966127 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966131 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966143 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966147 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966158 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:55.966257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966162 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78952e0-14c3-4811-8515-dc14aa6d7449" containerName="cleanup" Oct 30 14:30:55.966556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966299 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab9d46c-29a1-479a-a477-ae3f53404b91" containerName="cleanup" Oct 30 14:30:55.966556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966311 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f21dc7-a231-42c9-a073-19ad3fb9f22b" containerName="cleanup" Oct 30 14:30:55.966556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966320 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5baa41cf-dc7b-4386-a579-92f33e4ae890" containerName="cleanup" Oct 30 14:30:55.966556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966331 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1e643a-9a04-4a09-8d90-ccccdd02977c" containerName="cleanup" Oct 30 14:30:55.966556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966339 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a085f71-14d2-4210-b228-abd9eb70f463" containerName="cleanup" Oct 30 14:30:55.966556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966348 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb33c94-0194-4fc3-b8cf-ecfafa9ad28b" containerName="cleanup" Oct 30 14:30:55.966556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.966359 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a225fb-a633-426a-8a53-f3214aa77efb" containerName="cleanup" Oct 30 14:30:55.972034 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.972013 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:55.989283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.989260 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks"] Oct 30 14:30:55.997059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.997037 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:30:55.997059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:55.997042 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:30:56.112072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.112032 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.112241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.112183 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crf68\" (UniqueName: \"kubernetes.io/projected/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-kube-api-access-crf68\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.112285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.112254 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.213673 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.213643 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.213819 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.213726 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crf68\" (UniqueName: \"kubernetes.io/projected/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-kube-api-access-crf68\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.213819 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.213772 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.214471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.214447 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.216161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.216112 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.240284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.240262 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crf68\" (UniqueName: \"kubernetes.io/projected/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-kube-api-access-crf68\") pod \"maas-api-revoke-sub-69dcf5e6-95zks\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.287989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.287964 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:56.609538 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:56.604383 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks"] Oct 30 14:30:57.109711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:57.109640 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" event={"ID":"1b20ce00-1dca-4657-8a61-2bf8e2cffe09","Type":"ContainerStarted","Data":"e12c1112da623dae67c006c5728f7ddfe5d9dc331e3f0f490a44795f6e8948c6"} Oct 30 14:30:57.110035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:57.109729 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" event={"ID":"1b20ce00-1dca-4657-8a61-2bf8e2cffe09","Type":"ContainerStarted","Data":"3fc4c616517f59de413cc75365599d9c69bc7c7e5ece61e53cf66888a72e9c90"} Oct 30 14:30:57.157427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:57.157381 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" podStartSLOduration=2.1573692429999998 podStartE2EDuration="2.157369243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:30:57.154690715 +0000 UTC m=+2486.955196160" watchObservedRunningTime="2026-10-30 14:30:57.157369243 +0000 UTC m=+2486.957874686" Oct 30 14:30:57.801691 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:57.801584 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" podUID="e465ab52-9753-43f7-8b53-50e20ce58401" containerName="payload-pre-processing" containerID="cri-o://dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357" gracePeriod=25 Oct 30 14:30:58.119027 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:58.118951 5824 generic.go:356] "Generic (PLEG): container finished" podID="1b20ce00-1dca-4657-8a61-2bf8e2cffe09" containerID="e12c1112da623dae67c006c5728f7ddfe5d9dc331e3f0f490a44795f6e8948c6" exitCode=0 Oct 30 14:30:58.119417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:58.119039 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" event={"ID":"1b20ce00-1dca-4657-8a61-2bf8e2cffe09","Type":"ContainerDied","Data":"e12c1112da623dae67c006c5728f7ddfe5d9dc331e3f0f490a44795f6e8948c6"} Oct 30 14:30:59.413311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.405666 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:30:59.488788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.488752 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-xks-ca\") pod \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " Oct 30 14:30:59.489003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.488824 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-kube-api-access-crf68\" (UniqueName: \"kubernetes.io/projected/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-kube-api-access-crf68\") pod \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " Oct 30 14:30:59.489003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.488901 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-openshift-ca\") pod \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\" (UID: \"1b20ce00-1dca-4657-8a61-2bf8e2cffe09\") " Oct 30 14:30:59.489910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.489875 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-openshift-ca" pod "1b20ce00-1dca-4657-8a61-2bf8e2cffe09" (UID: "1b20ce00-1dca-4657-8a61-2bf8e2cffe09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:30:59.491415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.491389 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:59.494008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.493982 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-kube-api-access-crf68" pod "1b20ce00-1dca-4657-8a61-2bf8e2cffe09" (UID: "1b20ce00-1dca-4657-8a61-2bf8e2cffe09"). InnerVolumeSpecName "kube-api-access-crf68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:30:59.494142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.494109 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-xks-ca" pod "1b20ce00-1dca-4657-8a61-2bf8e2cffe09" (UID: "1b20ce00-1dca-4657-8a61-2bf8e2cffe09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:30:59.592664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.592633 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crf68\" (UniqueName: \"kubernetes.io/projected/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-kube-api-access-crf68\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:30:59.592664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:30:59.592660 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b20ce00-1dca-4657-8a61-2bf8e2cffe09-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:00.135931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:00.135903 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" Oct 30 14:31:00.135931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:00.135918 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks" event={"ID":"1b20ce00-1dca-4657-8a61-2bf8e2cffe09","Type":"ContainerDied","Data":"3fc4c616517f59de413cc75365599d9c69bc7c7e5ece61e53cf66888a72e9c90"} Oct 30 14:31:00.136206 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:00.135974 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc4c616517f59de413cc75365599d9c69bc7c7e5ece61e53cf66888a72e9c90" Oct 30 14:31:03.586566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:03.586523 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" podUID="e465ab52-9753-43f7-8b53-50e20ce58401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:31:03.643798 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:31:03.643761 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:31:07.937131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:07.937095 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph"] Oct 30 14:31:07.937934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:07.937916 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1b20ce00-1dca-4657-8a61-2bf8e2cffe09" containerName="revoke-keys" Oct 30 14:31:07.937934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:07.937934 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b20ce00-1dca-4657-8a61-2bf8e2cffe09" containerName="revoke-keys" Oct 30 14:31:07.938091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:07.938081 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b20ce00-1dca-4657-8a61-2bf8e2cffe09" containerName="revoke-keys" Oct 30 14:31:07.942391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:07.942372 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:07.961553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:07.961528 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph"] Oct 30 14:31:07.969488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:07.969468 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:07.969666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:07.969486 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:31:08.042161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.042130 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.042391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.042342 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.042568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.042551 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjfw\" (UniqueName: \"kubernetes.io/projected/72a56a75-bf25-4803-a2c0-0b24abb0ff94-kube-api-access-msjfw\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.144291 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.144250 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.144503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.144423 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.144641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.144625 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjfw\" (UniqueName: \"kubernetes.io/projected/72a56a75-bf25-4803-a2c0-0b24abb0ff94-kube-api-access-msjfw\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.145008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.144989 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.146917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.146899 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.169943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.169903 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjfw\" (UniqueName: \"kubernetes.io/projected/72a56a75-bf25-4803-a2c0-0b24abb0ff94-kube-api-access-msjfw\") pod \"maas-api-revoke-sub-0db98211-dhhph\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.256903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.256828 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:08.577793 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:31:08.577766 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a56a75_bf25_4803_a2c0_0b24abb0ff94.slice/crio-c87da7cd239dc424759f9f2d19c6b6e83815afb440cd64d7ac84b009728ac236 WatchSource:0}: Error finding container c87da7cd239dc424759f9f2d19c6b6e83815afb440cd64d7ac84b009728ac236: Status 404 returned error can't find the container with id c87da7cd239dc424759f9f2d19c6b6e83815afb440cd64d7ac84b009728ac236 Oct 30 14:31:08.578663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:08.578640 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph"] Oct 30 14:31:09.211826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:09.211739 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" event={"ID":"72a56a75-bf25-4803-a2c0-0b24abb0ff94","Type":"ContainerStarted","Data":"00372229bc1fbe1992a542df954a42e6d837eac0f94d29619e23c6fbcbe6e298"} Oct 30 14:31:09.212329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:09.211834 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" event={"ID":"72a56a75-bf25-4803-a2c0-0b24abb0ff94","Type":"ContainerStarted","Data":"c87da7cd239dc424759f9f2d19c6b6e83815afb440cd64d7ac84b009728ac236"} Oct 30 14:31:09.258051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:09.257993 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" podStartSLOduration=2.257979231 podStartE2EDuration="2.257979231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:09.25677374 +0000 UTC m=+2499.057279185" watchObservedRunningTime="2026-10-30 14:31:09.257979231 +0000 UTC m=+2499.058484675" Oct 30 14:31:10.220478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:10.220443 5824 generic.go:356] "Generic (PLEG): container finished" podID="72a56a75-bf25-4803-a2c0-0b24abb0ff94" containerID="00372229bc1fbe1992a542df954a42e6d837eac0f94d29619e23c6fbcbe6e298" exitCode=0 Oct 30 14:31:10.220876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:10.220523 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" event={"ID":"72a56a75-bf25-4803-a2c0-0b24abb0ff94","Type":"ContainerDied","Data":"00372229bc1fbe1992a542df954a42e6d837eac0f94d29619e23c6fbcbe6e298"} Oct 30 14:31:11.413231 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.413182 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:11.607025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.606947 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-openshift-ca\") pod \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " Oct 30 14:31:11.607220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.607136 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a56a75-bf25-4803-a2c0-0b24abb0ff94-kube-api-access-msjfw\" (UniqueName: \"kubernetes.io/projected/72a56a75-bf25-4803-a2c0-0b24abb0ff94-kube-api-access-msjfw\") pod \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " Oct 30 14:31:11.607310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.607223 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-xks-ca\") pod \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\" (UID: \"72a56a75-bf25-4803-a2c0-0b24abb0ff94\") " Oct 30 14:31:11.607395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.607367 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-openshift-ca" pod "72a56a75-bf25-4803-a2c0-0b24abb0ff94" (UID: "72a56a75-bf25-4803-a2c0-0b24abb0ff94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:11.607821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.607797 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:11.609238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.609207 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a56a75-bf25-4803-a2c0-0b24abb0ff94-kube-api-access-msjfw" pod "72a56a75-bf25-4803-a2c0-0b24abb0ff94" (UID: "72a56a75-bf25-4803-a2c0-0b24abb0ff94"). InnerVolumeSpecName "kube-api-access-msjfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:11.609425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.609401 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-xks-ca" pod "72a56a75-bf25-4803-a2c0-0b24abb0ff94" (UID: "72a56a75-bf25-4803-a2c0-0b24abb0ff94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:11.709472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.709438 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjfw\" (UniqueName: \"kubernetes.io/projected/72a56a75-bf25-4803-a2c0-0b24abb0ff94-kube-api-access-msjfw\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:11.709472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:11.709464 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a56a75-bf25-4803-a2c0-0b24abb0ff94-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:12.236047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:12.236017 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" Oct 30 14:31:12.236241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:12.236045 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph" event={"ID":"72a56a75-bf25-4803-a2c0-0b24abb0ff94","Type":"ContainerDied","Data":"c87da7cd239dc424759f9f2d19c6b6e83815afb440cd64d7ac84b009728ac236"} Oct 30 14:31:12.236241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:12.236096 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87da7cd239dc424759f9f2d19c6b6e83815afb440cd64d7ac84b009728ac236" Oct 30 14:31:13.586068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:13.586026 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" podUID="e465ab52-9753-43f7-8b53-50e20ce58401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:31:18.140531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.140494 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:31:18.230118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.230075 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e465ab52-9753-43f7-8b53-50e20ce58401-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e465ab52-9753-43f7-8b53-50e20ce58401-plugins-config-volume\") pod \"e465ab52-9753-43f7-8b53-50e20ce58401\" (UID: \"e465ab52-9753-43f7-8b53-50e20ce58401\") " Oct 30 14:31:18.230392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.230372 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e465ab52-9753-43f7-8b53-50e20ce58401-kube-api-access-6j2pq\" (UniqueName: \"kubernetes.io/projected/e465ab52-9753-43f7-8b53-50e20ce58401-kube-api-access-6j2pq\") pod \"e465ab52-9753-43f7-8b53-50e20ce58401\" (UID: \"e465ab52-9753-43f7-8b53-50e20ce58401\") " Oct 30 14:31:18.230514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.230487 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e465ab52-9753-43f7-8b53-50e20ce58401-plugins-config-volume" pod "e465ab52-9753-43f7-8b53-50e20ce58401" (UID: "e465ab52-9753-43f7-8b53-50e20ce58401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:18.231144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.231118 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e465ab52-9753-43f7-8b53-50e20ce58401-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:18.232596 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.232564 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e465ab52-9753-43f7-8b53-50e20ce58401-kube-api-access-6j2pq" pod "e465ab52-9753-43f7-8b53-50e20ce58401" (UID: "e465ab52-9753-43f7-8b53-50e20ce58401"). InnerVolumeSpecName "kube-api-access-6j2pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:18.284436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.284402 5824 generic.go:356] "Generic (PLEG): container finished" podID="e465ab52-9753-43f7-8b53-50e20ce58401" containerID="dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357" exitCode=0 Oct 30 14:31:18.284590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.284448 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" Oct 30 14:31:18.284590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.284487 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" event={"ID":"e465ab52-9753-43f7-8b53-50e20ce58401","Type":"ContainerDied","Data":"dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357"} Oct 30 14:31:18.284590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.284548 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr" event={"ID":"e465ab52-9753-43f7-8b53-50e20ce58401","Type":"ContainerDied","Data":"28070cd35aa8b87759ad40217aae913ed71a7e4a099cfbf0d46b317fd2e0b50f"} Oct 30 14:31:18.284590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.284575 5824 scope.go:122] "RemoveContainer" containerID="dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357" Oct 30 14:31:18.296021 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.296001 5824 scope.go:122] "RemoveContainer" containerID="dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357" Oct 30 14:31:18.296332 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:31:18.296308 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357\": container with ID starting with dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357 not found: ID does not exist" containerID="dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357" Oct 30 14:31:18.296419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.296339 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357"} err="failed to get container status \"dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357\": rpc error: code = NotFound desc = could not find container \"dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357\": container with ID starting with dfca503841aad6c26ab83ef1f615919da789f7e55f912ce39999ecd2ef720357 not found: ID does not exist" Oct 30 14:31:18.332640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.332588 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j2pq\" (UniqueName: \"kubernetes.io/projected/e465ab52-9753-43f7-8b53-50e20ce58401-kube-api-access-6j2pq\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:18.370539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.370443 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr"] Oct 30 14:31:18.376059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:18.376031 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d15602-86cf9656c5-cr7rr"] Oct 30 14:31:19.664096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.664053 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx"] Oct 30 14:31:19.664926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.664905 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e465ab52-9753-43f7-8b53-50e20ce58401" containerName="payload-pre-processing" Oct 30 14:31:19.664926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.664926 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e465ab52-9753-43f7-8b53-50e20ce58401" containerName="payload-pre-processing" Oct 30 14:31:19.665113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.664939 5824 cpu_manager.go:414] "containerMap: removing container" podUID="72a56a75-bf25-4803-a2c0-0b24abb0ff94" containerName="revoke-keys" Oct 30 14:31:19.665113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.664943 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a56a75-bf25-4803-a2c0-0b24abb0ff94" containerName="revoke-keys" Oct 30 14:31:19.665113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.665081 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e465ab52-9753-43f7-8b53-50e20ce58401" containerName="payload-pre-processing" Oct 30 14:31:19.665113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.665093 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a56a75-bf25-4803-a2c0-0b24abb0ff94" containerName="revoke-keys" Oct 30 14:31:19.672213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.672175 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.683601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.683574 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx"] Oct 30 14:31:19.763368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.763329 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.763542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.763460 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzrw\" (UniqueName: \"kubernetes.io/projected/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-kube-api-access-qjzrw\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.763651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.763623 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.769430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.769407 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:31:19.769430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.769421 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:31:19.864813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.864780 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.864996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.864850 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzrw\" (UniqueName: \"kubernetes.io/projected/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-kube-api-access-qjzrw\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.864996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.864922 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.865674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.865649 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.867077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.867058 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.891608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.891573 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzrw\" (UniqueName: \"kubernetes.io/projected/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-kube-api-access-qjzrw\") pod \"maas-api-revoke-keys-e2e-inf-a-409364-6fmpx\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:19.989748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:19.989718 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:20.310555 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:31:20.310518 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9c632e_a54c_40ed_8b83_3f86509c9ae4.slice/crio-5d87baff1f5e5629be4a13cf945a76a13adf5b41c0df6ca8b14f18fb493c289c WatchSource:0}: Error finding container 5d87baff1f5e5629be4a13cf945a76a13adf5b41c0df6ca8b14f18fb493c289c: Status 404 returned error can't find the container with id 5d87baff1f5e5629be4a13cf945a76a13adf5b41c0df6ca8b14f18fb493c289c Oct 30 14:31:20.325486 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:20.325455 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e465ab52-9753-43f7-8b53-50e20ce58401" path="/var/lib/kubelet/pods/e465ab52-9753-43f7-8b53-50e20ce58401/volumes" Oct 30 14:31:20.325919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:20.325893 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx"] Oct 30 14:31:21.312130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:21.312093 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" event={"ID":"6c9c632e-a54c-40ed-8b83-3f86509c9ae4","Type":"ContainerStarted","Data":"b07fa1352fa03665596164841ae943f0bed6b1f2ec23f6deb142de687c6e4cea"} Oct 30 14:31:21.312549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:21.312155 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" event={"ID":"6c9c632e-a54c-40ed-8b83-3f86509c9ae4","Type":"ContainerStarted","Data":"5d87baff1f5e5629be4a13cf945a76a13adf5b41c0df6ca8b14f18fb493c289c"} Oct 30 14:31:21.356946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:21.356891 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" podStartSLOduration=2.356873031 podStartE2EDuration="2.356873031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:31:21.3555265 +0000 UTC m=+2511.156031947" watchObservedRunningTime="2026-10-30 14:31:21.356873031 +0000 UTC m=+2511.157378476" Oct 30 14:31:22.331002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:22.330964 5824 generic.go:356] "Generic (PLEG): container finished" podID="6c9c632e-a54c-40ed-8b83-3f86509c9ae4" containerID="b07fa1352fa03665596164841ae943f0bed6b1f2ec23f6deb142de687c6e4cea" exitCode=0 Oct 30 14:31:22.331481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:22.331048 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" event={"ID":"6c9c632e-a54c-40ed-8b83-3f86509c9ae4","Type":"ContainerDied","Data":"b07fa1352fa03665596164841ae943f0bed6b1f2ec23f6deb142de687c6e4cea"} Oct 30 14:31:23.508052 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.508023 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:23.624573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.624490 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-xks-ca\") pod \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " Oct 30 14:31:23.624573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.624539 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-openshift-ca\") pod \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " Oct 30 14:31:23.624940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.624617 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-kube-api-access-qjzrw\" (UniqueName: \"kubernetes.io/projected/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-kube-api-access-qjzrw\") pod \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\" (UID: \"6c9c632e-a54c-40ed-8b83-3f86509c9ae4\") " Oct 30 14:31:23.624940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.624905 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-openshift-ca" pod "6c9c632e-a54c-40ed-8b83-3f86509c9ae4" (UID: "6c9c632e-a54c-40ed-8b83-3f86509c9ae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:23.625255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.625236 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:23.626645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.626621 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-xks-ca" pod "6c9c632e-a54c-40ed-8b83-3f86509c9ae4" (UID: "6c9c632e-a54c-40ed-8b83-3f86509c9ae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:31:23.626762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.626745 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-kube-api-access-qjzrw" pod "6c9c632e-a54c-40ed-8b83-3f86509c9ae4" (UID: "6c9c632e-a54c-40ed-8b83-3f86509c9ae4"). InnerVolumeSpecName "kube-api-access-qjzrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:23.727074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.727042 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:23.727074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:23.727070 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzrw\" (UniqueName: \"kubernetes.io/projected/6c9c632e-a54c-40ed-8b83-3f86509c9ae4-kube-api-access-qjzrw\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:24.346674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:24.346650 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" Oct 30 14:31:24.346840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:24.346646 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx" event={"ID":"6c9c632e-a54c-40ed-8b83-3f86509c9ae4","Type":"ContainerDied","Data":"5d87baff1f5e5629be4a13cf945a76a13adf5b41c0df6ca8b14f18fb493c289c"} Oct 30 14:31:24.346840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:24.346767 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d87baff1f5e5629be4a13cf945a76a13adf5b41c0df6ca8b14f18fb493c289c" Oct 30 14:31:26.655233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:26.655168 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj"] Oct 30 14:31:26.655795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:26.655418 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" containerID="cri-o://0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207" gracePeriod=30 Oct 30 14:31:29.340170 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:29.340137 5824 patch_prober.go:29] interesting pod/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:31:29.340603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:29.340220 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:31:30.490909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:30.490873 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh"] Oct 30 14:31:31.951638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:31.951574 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:31:32.101944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.101896 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istiod-ca-cert\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.102225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.101984 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-credential-socket\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.102225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.102071 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-kube-api-access-tmqrj\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-kube-api-access-tmqrj\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.102225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.102153 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-token\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-token\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.102476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.102272 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-certs\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.102476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.102344 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-socket\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.102619 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.102487 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-podinfo\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.106327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.106295 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-envoy\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.106483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.106348 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-data\") pod \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\" (UID: \"da87cb61-ccfa-49c2-9c39-cd6583a00d3c\") " Oct 30 14:31:32.107370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.107339 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-data" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:32.108999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.108764 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istiod-ca-cert" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:32.109536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.109164 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-socket" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:32.109827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.109797 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-credential-socket" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:32.110916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.110880 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-certs" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:32.112628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.112587 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-kube-api-access-tmqrj" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "kube-api-access-tmqrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:32.115378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.115341 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-envoy" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:32.122864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.122830 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-token" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:32.123140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.123114 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-podinfo" pod "da87cb61-ccfa-49c2-9c39-cd6583a00d3c" (UID: "da87cb61-ccfa-49c2-9c39-cd6583a00d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:31:32.208228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208182 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqrj\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-kube-api-access-tmqrj\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.208228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208225 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.208412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208239 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.208412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208248 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.208412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208259 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.208412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208267 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.208412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208275 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.208412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208282 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.208412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.208293 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da87cb61-ccfa-49c2-9c39-cd6583a00d3c-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:32.419473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.419438 5824 generic.go:356] "Generic (PLEG): container finished" podID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerID="0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207" exitCode=0 Oct 30 14:31:32.419712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.419486 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" Oct 30 14:31:32.419712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.419511 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" event={"ID":"da87cb61-ccfa-49c2-9c39-cd6583a00d3c","Type":"ContainerDied","Data":"0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207"} Oct 30 14:31:32.419712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.419566 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj" event={"ID":"da87cb61-ccfa-49c2-9c39-cd6583a00d3c","Type":"ContainerDied","Data":"036bf5cff7a1e7930cf25a4bb1fd67f9bb7c26e2804be8369efdb169d79fa05f"} Oct 30 14:31:32.419712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.419578 5824 scope.go:122] "RemoveContainer" containerID="0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207" Oct 30 14:31:32.430898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.430878 5824 scope.go:122] "RemoveContainer" containerID="0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207" Oct 30 14:31:32.431206 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:31:32.431170 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207\": container with ID starting with 0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207 not found: ID does not exist" containerID="0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207" Oct 30 14:31:32.431296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.431213 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207"} err="failed to get container status \"0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207\": rpc error: code = NotFound desc = could not find container \"0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207\": container with ID starting with 0f95ff2d8debf30643019ca4d214a847164b3b240098edbc99668e62fbb86207 not found: ID does not exist" Oct 30 14:31:32.480354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.480284 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj"] Oct 30 14:31:32.492961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:32.492932 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d15602-openshift-default-6b94954795-nvxwj"] Oct 30 14:31:33.645592 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:31:33.645559 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:31:34.319092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:34.319059 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" path="/var/lib/kubelet/pods/da87cb61-ccfa-49c2-9c39-cd6583a00d3c/volumes" Oct 30 14:31:35.492010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:35.491974 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" podUID="e55fba24-dca9-4b03-b9a0-645ee11072a9" containerName="payload-pre-processing" containerID="cri-o://93dfbe084b7a33ea9cd8df7afc4a0ab571b5731feeb5e3cbf33213e9f00da6b3" gracePeriod=25 Oct 30 14:31:37.174095 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:37.174059 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" podUID="e55fba24-dca9-4b03-b9a0-645ee11072a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:31:37.654434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:37.654354 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf"] Oct 30 14:31:42.655715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:42.655472 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" podUID="6a948cff-ab50-424c-bc53-b5d16ff54c84" containerName="payload-pre-processing" containerID="cri-o://e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d" gracePeriod=25 Oct 30 14:31:44.117444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:44.117407 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv"] Oct 30 14:31:44.117861 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:44.117617 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" containerID="cri-o://0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38" gracePeriod=30 Oct 30 14:31:45.904672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:45.904612 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" podUID="6a948cff-ab50-424c-bc53-b5d16ff54c84" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:31:47.174053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:47.174011 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" podUID="e55fba24-dca9-4b03-b9a0-645ee11072a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:31:49.391991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.391931 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:31:49.506855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.506816 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-certs\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.506870 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-data\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.506965 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-credential-socket\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507026 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-socket\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507084 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-token\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-token\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507127 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-kube-api-access-t5bqs\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-kube-api-access-t5bqs\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507293 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-data" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:49.507394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507325 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-podinfo\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507395 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-envoy\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507506 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istiod-ca-cert\") pod \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\" (UID: \"7a2c31c7-c84d-4e01-a1db-53b637f7882d\") " Oct 30 14:31:49.507990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507696 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-socket" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:49.507990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507734 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-certs" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:49.507990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.507969 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-credential-socket" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:49.508235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.508083 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istiod-ca-cert" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:49.508366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.508343 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.508533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.508517 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.508652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.508639 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.508748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.508735 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.508846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.508833 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.510074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.509905 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-kube-api-access-t5bqs" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "kube-api-access-t5bqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:49.510074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.509975 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-token" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:49.510439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.510414 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-envoy" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:49.510611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.510587 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-podinfo" pod "7a2c31c7-c84d-4e01-a1db-53b637f7882d" (UID: "7a2c31c7-c84d-4e01-a1db-53b637f7882d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:31:49.557286 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.557259 5824 generic.go:356] "Generic (PLEG): container finished" podID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerID="0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38" exitCode=0 Oct 30 14:31:49.557411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.557299 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" Oct 30 14:31:49.557411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.557328 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" event={"ID":"7a2c31c7-c84d-4e01-a1db-53b637f7882d","Type":"ContainerDied","Data":"0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38"} Oct 30 14:31:49.557411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.557408 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv" event={"ID":"7a2c31c7-c84d-4e01-a1db-53b637f7882d","Type":"ContainerDied","Data":"f2d00277defe51d9d6bbccff6e79d5973d87b22028ce1446b44a933b4814f0fe"} Oct 30 14:31:49.557516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.557437 5824 scope.go:122] "RemoveContainer" containerID="0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38" Oct 30 14:31:49.576016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.575990 5824 scope.go:122] "RemoveContainer" containerID="0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38" Oct 30 14:31:49.576380 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:31:49.576354 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38\": container with ID starting with 0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38 not found: ID does not exist" containerID="0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38" Oct 30 14:31:49.576469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.576393 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38"} err="failed to get container status \"0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38\": rpc error: code = NotFound desc = could not find container \"0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38\": container with ID starting with 0660c9305c9d7961d5f421f353fe61fe5e86f12e14f0484014ac67c243f22b38 not found: ID does not exist" Oct 30 14:31:49.609885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.609857 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.609885 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.609884 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5bqs\" (UniqueName: \"kubernetes.io/projected/7a2c31c7-c84d-4e01-a1db-53b637f7882d-kube-api-access-t5bqs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.610091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.609895 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.610091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.609904 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2c31c7-c84d-4e01-a1db-53b637f7882d-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:49.621541 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.621502 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv"] Oct 30 14:31:49.628129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:49.628101 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00926c-openshift-default-5b8b9f5575-kg9dv"] Oct 30 14:31:50.320086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:50.320045 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" path="/var/lib/kubelet/pods/7a2c31c7-c84d-4e01-a1db-53b637f7882d/volumes" Oct 30 14:31:52.719707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:52.719626 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm"] Oct 30 14:31:52.720072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:52.719873 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" containerID="cri-o://0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6" gracePeriod=30 Oct 30 14:31:55.613626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.613572 5824 patch_prober.go:29] interesting pod/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:31:55.614054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.613644 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:31:55.614054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.613897 5824 generic.go:356] "Generic (PLEG): container finished" podID="e55fba24-dca9-4b03-b9a0-645ee11072a9" containerID="93dfbe084b7a33ea9cd8df7afc4a0ab571b5731feeb5e3cbf33213e9f00da6b3" exitCode=0 Oct 30 14:31:55.614054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.613966 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" event={"ID":"e55fba24-dca9-4b03-b9a0-645ee11072a9","Type":"ContainerDied","Data":"93dfbe084b7a33ea9cd8df7afc4a0ab571b5731feeb5e3cbf33213e9f00da6b3"} Oct 30 14:31:55.791115 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.791078 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:31:55.808339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.808312 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55fba24-dca9-4b03-b9a0-645ee11072a9-kube-api-access-ffsfb\" (UniqueName: \"kubernetes.io/projected/e55fba24-dca9-4b03-b9a0-645ee11072a9-kube-api-access-ffsfb\") pod \"e55fba24-dca9-4b03-b9a0-645ee11072a9\" (UID: \"e55fba24-dca9-4b03-b9a0-645ee11072a9\") " Oct 30 14:31:55.808485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.808457 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55fba24-dca9-4b03-b9a0-645ee11072a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55fba24-dca9-4b03-b9a0-645ee11072a9-plugins-config-volume\") pod \"e55fba24-dca9-4b03-b9a0-645ee11072a9\" (UID: \"e55fba24-dca9-4b03-b9a0-645ee11072a9\") " Oct 30 14:31:55.808796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.808775 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55fba24-dca9-4b03-b9a0-645ee11072a9-plugins-config-volume" pod "e55fba24-dca9-4b03-b9a0-645ee11072a9" (UID: "e55fba24-dca9-4b03-b9a0-645ee11072a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:55.809013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.808995 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55fba24-dca9-4b03-b9a0-645ee11072a9-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:55.810643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.810609 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55fba24-dca9-4b03-b9a0-645ee11072a9-kube-api-access-ffsfb" pod "e55fba24-dca9-4b03-b9a0-645ee11072a9" (UID: "e55fba24-dca9-4b03-b9a0-645ee11072a9"). InnerVolumeSpecName "kube-api-access-ffsfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:55.898989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.898948 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" podUID="6a948cff-ab50-424c-bc53-b5d16ff54c84" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:31:55.910123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:55.910097 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffsfb\" (UniqueName: \"kubernetes.io/projected/e55fba24-dca9-4b03-b9a0-645ee11072a9-kube-api-access-ffsfb\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:56.624216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:56.624120 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" Oct 30 14:31:56.624216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:56.624144 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh" event={"ID":"e55fba24-dca9-4b03-b9a0-645ee11072a9","Type":"ContainerDied","Data":"86ec666df7279a5ed2d4fbaabc877724dbdb0c8cacc43df18122296447deb26f"} Oct 30 14:31:56.624748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:56.624230 5824 scope.go:122] "RemoveContainer" containerID="93dfbe084b7a33ea9cd8df7afc4a0ab571b5731feeb5e3cbf33213e9f00da6b3" Oct 30 14:31:56.685341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:56.685307 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh"] Oct 30 14:31:56.690349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:56.690317 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-409364-89dc78747-j29bh"] Oct 30 14:31:58.075884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.075860 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:31:58.151512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.151466 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-token\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-token\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.151811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.151792 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-data\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.153081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.152132 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-kube-api-access-jbj94\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-kube-api-access-jbj94\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.153081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.152249 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-envoy\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.153081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.152282 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-socket\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.153081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.152316 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-certs\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.153081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.152327 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-data" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:58.153081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.152452 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-podinfo\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.153503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.152889 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-socket" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:58.153503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.153054 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-certs" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:58.190259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.178787 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-credential-socket\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.190259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.178881 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e05408-2277-4bfd-a618-b8cbd2249911-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e05408-2277-4bfd-a618-b8cbd2249911-istiod-ca-cert\") pod \"a0e05408-2277-4bfd-a618-b8cbd2249911\" (UID: \"a0e05408-2277-4bfd-a618-b8cbd2249911\") " Oct 30 14:31:58.190259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.179822 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.190259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.179851 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.190259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.179866 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.190259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.180335 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e05408-2277-4bfd-a618-b8cbd2249911-istiod-ca-cert" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:31:58.190259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.180638 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-credential-socket" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:58.205452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.205404 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-token" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:58.222928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.222894 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-kube-api-access-jbj94" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "kube-api-access-jbj94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:31:58.223313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.223262 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-envoy" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:31:58.224976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.224935 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-podinfo" pod "a0e05408-2277-4bfd-a618-b8cbd2249911" (UID: "a0e05408-2277-4bfd-a618-b8cbd2249911"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:31:58.281410 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.281376 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.281410 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.281411 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e05408-2277-4bfd-a618-b8cbd2249911-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.281649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.281441 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.281649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.281456 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbj94\" (UniqueName: \"kubernetes.io/projected/a0e05408-2277-4bfd-a618-b8cbd2249911-kube-api-access-jbj94\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.281649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.281472 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.281649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.281485 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e05408-2277-4bfd-a618-b8cbd2249911-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:31:58.319690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.319621 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55fba24-dca9-4b03-b9a0-645ee11072a9" path="/var/lib/kubelet/pods/e55fba24-dca9-4b03-b9a0-645ee11072a9/volumes" Oct 30 14:31:58.643545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.643464 5824 generic.go:356] "Generic (PLEG): container finished" podID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerID="0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6" exitCode=0 Oct 30 14:31:58.643545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.643513 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" Oct 30 14:31:58.643743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.643554 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" event={"ID":"a0e05408-2277-4bfd-a618-b8cbd2249911","Type":"ContainerDied","Data":"0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6"} Oct 30 14:31:58.643743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.643609 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm" event={"ID":"a0e05408-2277-4bfd-a618-b8cbd2249911","Type":"ContainerDied","Data":"29a56e3635afbbb122c07f5b527770eb908f5650bc9c8599c575d0b6091e6a9b"} Oct 30 14:31:58.643743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.643637 5824 scope.go:122] "RemoveContainer" containerID="0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6" Oct 30 14:31:58.654773 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.654746 5824 scope.go:122] "RemoveContainer" containerID="0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6" Oct 30 14:31:58.655302 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:31:58.655057 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6\": container with ID starting with 0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6 not found: ID does not exist" containerID="0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6" Oct 30 14:31:58.655302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.655092 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6"} err="failed to get container status \"0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6\": rpc error: code = NotFound desc = could not find container \"0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6\": container with ID starting with 0f9e6bfd26fa17fbdc98a310dfefc9d1db452c332640477c5b729d67e57efaa6 not found: ID does not exist" Oct 30 14:31:58.716319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.716291 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm"] Oct 30 14:31:58.744428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:31:58.744390 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-93549e-openshift-default-5c77df5c4c-pb7fm"] Oct 30 14:32:00.319729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:00.319694 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" path="/var/lib/kubelet/pods/a0e05408-2277-4bfd-a618-b8cbd2249911/volumes" Oct 30 14:32:02.942891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:02.942866 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:32:03.059684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.059161 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a948cff-ab50-424c-bc53-b5d16ff54c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a948cff-ab50-424c-bc53-b5d16ff54c84-plugins-config-volume\") pod \"6a948cff-ab50-424c-bc53-b5d16ff54c84\" (UID: \"6a948cff-ab50-424c-bc53-b5d16ff54c84\") " Oct 30 14:32:03.059684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.059343 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a948cff-ab50-424c-bc53-b5d16ff54c84-kube-api-access-9pkgp\" (UniqueName: \"kubernetes.io/projected/6a948cff-ab50-424c-bc53-b5d16ff54c84-kube-api-access-9pkgp\") pod \"6a948cff-ab50-424c-bc53-b5d16ff54c84\" (UID: \"6a948cff-ab50-424c-bc53-b5d16ff54c84\") " Oct 30 14:32:03.086539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.086492 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a948cff-ab50-424c-bc53-b5d16ff54c84-plugins-config-volume" pod "6a948cff-ab50-424c-bc53-b5d16ff54c84" (UID: "6a948cff-ab50-424c-bc53-b5d16ff54c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:03.089774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.089741 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a948cff-ab50-424c-bc53-b5d16ff54c84-kube-api-access-9pkgp" pod "6a948cff-ab50-424c-bc53-b5d16ff54c84" (UID: "6a948cff-ab50-424c-bc53-b5d16ff54c84"). InnerVolumeSpecName "kube-api-access-9pkgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:03.162144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.162106 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a948cff-ab50-424c-bc53-b5d16ff54c84-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:03.162144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.162141 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkgp\" (UniqueName: \"kubernetes.io/projected/6a948cff-ab50-424c-bc53-b5d16ff54c84-kube-api-access-9pkgp\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:03.642243 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:32:03.642173 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:32:03.695502 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.695463 5824 generic.go:356] "Generic (PLEG): container finished" podID="6a948cff-ab50-424c-bc53-b5d16ff54c84" containerID="e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d" exitCode=0 Oct 30 14:32:03.695732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.695520 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" Oct 30 14:32:03.695732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.695554 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" event={"ID":"6a948cff-ab50-424c-bc53-b5d16ff54c84","Type":"ContainerDied","Data":"e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d"} Oct 30 14:32:03.695732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.695655 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf" event={"ID":"6a948cff-ab50-424c-bc53-b5d16ff54c84","Type":"ContainerDied","Data":"97fc121077555d6c6c895cd52ae69add56897d3864d5de9d7a3a197200cde964"} Oct 30 14:32:03.695732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.695691 5824 scope.go:122] "RemoveContainer" containerID="e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d" Oct 30 14:32:03.707643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.707621 5824 scope.go:122] "RemoveContainer" containerID="e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d" Oct 30 14:32:03.707938 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:32:03.707917 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d\": container with ID starting with e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d not found: ID does not exist" containerID="e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d" Oct 30 14:32:03.708013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.707949 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d"} err="failed to get container status \"e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d\": rpc error: code = NotFound desc = could not find container \"e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d\": container with ID starting with e66fb6d0fe3018874ef35c5d19e26234a59a8cd6018caf89620aaad7cf82912d not found: ID does not exist" Oct 30 14:32:03.760249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.760220 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf"] Oct 30 14:32:03.765051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:03.765027 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-93549e-779fdd989f-7cpgf"] Oct 30 14:32:04.318774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:04.318729 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a948cff-ab50-424c-bc53-b5d16ff54c84" path="/var/lib/kubelet/pods/6a948cff-ab50-424c-bc53-b5d16ff54c84/volumes" Oct 30 14:32:06.074010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:06.073969 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx"] Oct 30 14:32:06.087306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:06.087264 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-409364-6fmpx"] Oct 30 14:32:06.320208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:06.320169 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9c632e-a54c-40ed-8b83-3f86509c9ae4" path="/var/lib/kubelet/pods/6c9c632e-a54c-40ed-8b83-3f86509c9ae4/volumes" Oct 30 14:32:31.775014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:31.774981 5824 scope.go:122] "RemoveContainer" containerID="87684eba665e8c925437ce7b5438a3c8c87e00db3a3f9cb6b54689d8fb18c0ac" Oct 30 14:32:33.660371 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:32:33.660327 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:32:54.278772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:54.278700 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t"] Oct 30 14:32:54.279393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:54.278989 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" containerID="cri-o://5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db" gracePeriod=30 Oct 30 14:32:59.365102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.365069 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74"] Oct 30 14:32:59.365517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.365305 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" containerID="cri-o://175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d" gracePeriod=30 Oct 30 14:32:59.572471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.572445 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:32:59.696440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.696393 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-envoy\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.696659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.696464 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-credential-socket\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.696659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.696501 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-istio-token\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-istio-token\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.696659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.696535 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd369177-a300-45ac-a896-65e097ce54ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd369177-a300-45ac-a896-65e097ce54ed-istio-podinfo\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.696659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.696590 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd369177-a300-45ac-a896-65e097ce54ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd369177-a300-45ac-a896-65e097ce54ed-istiod-ca-cert\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.697009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.696779 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-credential-socket" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:59.697100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.697034 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-kube-api-access-f7bp7\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-kube-api-access-f7bp7\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.697182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.697171 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-socket\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.697299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.697264 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-certs\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.697382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.697301 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-data\") pod \"cd369177-a300-45ac-a896-65e097ce54ed\" (UID: \"cd369177-a300-45ac-a896-65e097ce54ed\") " Oct 30 14:32:59.698013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.697983 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.698617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.698588 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-certs" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:59.699926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.698606 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-data" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:59.699926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.698661 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-socket" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:59.699926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.698927 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd369177-a300-45ac-a896-65e097ce54ed-istiod-ca-cert" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:32:59.699926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.699497 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-istio-token" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:59.700288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.700260 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd369177-a300-45ac-a896-65e097ce54ed-istio-podinfo" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:32:59.701413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.701253 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-kube-api-access-f7bp7" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "kube-api-access-f7bp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:32:59.701413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.701329 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-envoy" pod "cd369177-a300-45ac-a896-65e097ce54ed" (UID: "cd369177-a300-45ac-a896-65e097ce54ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:32:59.799567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.799495 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.799567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.799523 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.799567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.799535 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd369177-a300-45ac-a896-65e097ce54ed-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.799567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.799547 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd369177-a300-45ac-a896-65e097ce54ed-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.799567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.799560 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bp7\" (UniqueName: \"kubernetes.io/projected/cd369177-a300-45ac-a896-65e097ce54ed-kube-api-access-f7bp7\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.799813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.799573 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.799813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.799586 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:32:59.799813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:32:59.799597 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd369177-a300-45ac-a896-65e097ce54ed-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:00.239310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.239275 5824 generic.go:356] "Generic (PLEG): container finished" podID="cd369177-a300-45ac-a896-65e097ce54ed" containerID="5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db" exitCode=0 Oct 30 14:33:00.239476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.239329 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" Oct 30 14:33:00.239476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.239368 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" event={"ID":"cd369177-a300-45ac-a896-65e097ce54ed","Type":"ContainerDied","Data":"5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db"} Oct 30 14:33:00.239476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.239420 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t" event={"ID":"cd369177-a300-45ac-a896-65e097ce54ed","Type":"ContainerDied","Data":"113472b30108226a2a13eb554992a30c0767db56f451118afd246ab4ef79cfdb"} Oct 30 14:33:00.239476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.239445 5824 scope.go:122] "RemoveContainer" containerID="5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db" Oct 30 14:33:00.251326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.251305 5824 scope.go:122] "RemoveContainer" containerID="5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db" Oct 30 14:33:00.251604 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:33:00.251585 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db\": container with ID starting with 5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db not found: ID does not exist" containerID="5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db" Oct 30 14:33:00.251684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.251613 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db"} err="failed to get container status \"5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db\": rpc error: code = NotFound desc = could not find container \"5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db\": container with ID starting with 5ecb0addf7609a1627bc64c744a122968679a4384d6a9afba62765f789ccb8db not found: ID does not exist" Oct 30 14:33:00.300429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.300404 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t"] Oct 30 14:33:00.319325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:00.319302 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f3c580-openshift-default-5856c7d869-4cx4t"] Oct 30 14:33:02.319089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:02.319055 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd369177-a300-45ac-a896-65e097ce54ed" path="/var/lib/kubelet/pods/cd369177-a300-45ac-a896-65e097ce54ed/volumes" Oct 30 14:33:02.889006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:02.888974 5824 patch_prober.go:29] interesting pod/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:33:02.889266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:02.889031 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:33:03.646222 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:33:03.646154 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:33:04.668637 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.668611 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:33:04.773394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773364 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-socket\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.773607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773416 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-certs\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.773607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773517 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-kube-api-access-r5l9m\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-kube-api-access-r5l9m\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.773607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773548 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-envoy\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.773805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773646 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-data\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.773805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773673 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-socket" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:04.773805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773752 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-podinfo\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.773805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773786 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-certs" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:04.774025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773809 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istiod-ca-cert\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.774025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773836 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-credential-socket\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.774025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.773881 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-token\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-token\") pod \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\" (UID: \"97e049a0-9ed9-49c5-83bf-54f76f39ee44\") " Oct 30 14:33:04.774292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.774266 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istiod-ca-cert" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:04.774645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.774615 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-data" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:04.774980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.774932 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-credential-socket" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:04.775296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.775184 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:04.775296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.775232 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:04.775296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.775248 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:04.775296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.775262 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:04.775296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.775280 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:04.776794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.776458 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-kube-api-access-r5l9m" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "kube-api-access-r5l9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:04.777028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.776991 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-token" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:04.778516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.778492 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-podinfo" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:33:04.778646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.778608 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-envoy" pod "97e049a0-9ed9-49c5-83bf-54f76f39ee44" (UID: "97e049a0-9ed9-49c5-83bf-54f76f39ee44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:04.877546 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.877508 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:04.877546 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.877546 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:04.877751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.877561 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5l9m\" (UniqueName: \"kubernetes.io/projected/97e049a0-9ed9-49c5-83bf-54f76f39ee44-kube-api-access-r5l9m\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:04.877751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:04.877575 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e049a0-9ed9-49c5-83bf-54f76f39ee44-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:05.254171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.254132 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65"] Oct 30 14:33:05.284060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.284024 5824 generic.go:356] "Generic (PLEG): container finished" podID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerID="175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d" exitCode=0 Oct 30 14:33:05.284312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.284095 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" Oct 30 14:33:05.284312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.284119 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" event={"ID":"97e049a0-9ed9-49c5-83bf-54f76f39ee44","Type":"ContainerDied","Data":"175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d"} Oct 30 14:33:05.284312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.284201 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74" event={"ID":"97e049a0-9ed9-49c5-83bf-54f76f39ee44","Type":"ContainerDied","Data":"0f473dbf0b1c5b557b55cf737a68205edd421eee7f5ef9398ed93dcb549e368e"} Oct 30 14:33:05.284312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.284238 5824 scope.go:122] "RemoveContainer" containerID="175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d" Oct 30 14:33:05.291871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.291846 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n"] Oct 30 14:33:05.300430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.300410 5824 scope.go:122] "RemoveContainer" containerID="175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d" Oct 30 14:33:05.300736 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:33:05.300696 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d\": container with ID starting with 175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d not found: ID does not exist" containerID="175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d" Oct 30 14:33:05.300791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.300747 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d"} err="failed to get container status \"175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d\": rpc error: code = NotFound desc = could not find container \"175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d\": container with ID starting with 175c11cb666dd143ef9ace9c171a40aeb8c286c395daa2a1dc72e8541cc3ab6d not found: ID does not exist" Oct 30 14:33:05.393477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.393439 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74"] Oct 30 14:33:05.400109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:05.400080 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a7bb0c-openshift-default-86bfdfb8bf-zdx74"] Oct 30 14:33:06.320452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:06.320413 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" path="/var/lib/kubelet/pods/97e049a0-9ed9-49c5-83bf-54f76f39ee44/volumes" Oct 30 14:33:10.255282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:10.255239 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" podUID="5a064383-9f17-459d-b2d6-0600806167fd" containerName="payload-processing" containerID="cri-o://ba8418417d19b371a43d712ba6a3ed9bbe06be6d7957e23d12603a0efa0d4ca6" gracePeriod=25 Oct 30 14:33:10.292821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:10.292783 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" podUID="0fcdf182-ec74-4302-9fbe-1a9068aa263c" containerName="payload-pre-processing" containerID="cri-o://a8d094220567b5f84043a2bd53cc5a75bcc78c13dc9ff34749e78464dcc2e506" gracePeriod=25 Oct 30 14:33:12.279287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:12.279177 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq"] Oct 30 14:33:12.279801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:12.279771 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" containerID="cri-o://2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02" gracePeriod=30 Oct 30 14:33:15.472703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:15.472655 5824 patch_prober.go:29] interesting pod/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:33:15.473226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:15.472741 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:33:17.576609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.576577 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:33:17.766063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766021 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-data\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766151 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-credential-socket\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766231 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-envoy\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766306 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-socket\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766343 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-token\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-token\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766342 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-data" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:17.766586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766460 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-credential-socket" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:17.766586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766544 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-certs\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766596 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-podinfo\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766614 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-socket" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:17.766705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766623 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istiod-ca-cert\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766675 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-kube-api-access-zlk76\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-kube-api-access-zlk76\") pod \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\" (UID: \"159f5cf3-e422-4f1a-9769-cb20b007d4b0\") " Oct 30 14:33:17.766915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766845 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-certs" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:17.766956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.766921 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istiod-ca-cert" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:17.767418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.767394 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:17.767418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.767416 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:17.767418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.767424 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:17.767670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.767433 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:17.767670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.767441 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:17.768707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.768683 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-token" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:17.768829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.768726 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-podinfo" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:33:17.769114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.769090 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-envoy" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:17.769218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.769158 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-kube-api-access-zlk76" pod "159f5cf3-e422-4f1a-9769-cb20b007d4b0" (UID: "159f5cf3-e422-4f1a-9769-cb20b007d4b0"). InnerVolumeSpecName "kube-api-access-zlk76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:17.868674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.868587 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:17.868674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.868621 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:17.868674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.868635 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlk76\" (UniqueName: \"kubernetes.io/projected/159f5cf3-e422-4f1a-9769-cb20b007d4b0-kube-api-access-zlk76\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:17.868674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:17.868649 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159f5cf3-e422-4f1a-9769-cb20b007d4b0-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:18.414484 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.414450 5824 generic.go:356] "Generic (PLEG): container finished" podID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerID="2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02" exitCode=0 Oct 30 14:33:18.414639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.414504 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" Oct 30 14:33:18.414639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.414530 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" event={"ID":"159f5cf3-e422-4f1a-9769-cb20b007d4b0","Type":"ContainerDied","Data":"2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02"} Oct 30 14:33:18.414639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.414606 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq" event={"ID":"159f5cf3-e422-4f1a-9769-cb20b007d4b0","Type":"ContainerDied","Data":"04a0fb5518e0a27f6d3ea297c1b2047501660a7ba52cc450327b7294636c0b96"} Oct 30 14:33:18.414639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.414628 5824 scope.go:122] "RemoveContainer" containerID="2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02" Oct 30 14:33:18.428748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.428727 5824 scope.go:122] "RemoveContainer" containerID="2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02" Oct 30 14:33:18.429049 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:33:18.429032 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02\": container with ID starting with 2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02 not found: ID does not exist" containerID="2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02" Oct 30 14:33:18.429088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.429056 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02"} err="failed to get container status \"2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02\": rpc error: code = NotFound desc = could not find container \"2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02\": container with ID starting with 2e9edf6a53ee7a9bb1857ed0f76dd2c7d4fa8bea34222b73c7e46470f1357a02 not found: ID does not exist" Oct 30 14:33:18.497940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.497910 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq"] Oct 30 14:33:18.504732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:18.504711 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fde75d-openshift-default-6fb6875ff-khlmq"] Oct 30 14:33:20.117547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:20.117515 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" podUID="5a064383-9f17-459d-b2d6-0600806167fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:33:20.117547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:20.117535 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" podUID="0fcdf182-ec74-4302-9fbe-1a9068aa263c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:33:20.319656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:20.319621 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" path="/var/lib/kubelet/pods/159f5cf3-e422-4f1a-9769-cb20b007d4b0/volumes" Oct 30 14:33:30.116823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.116783 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" podUID="0fcdf182-ec74-4302-9fbe-1a9068aa263c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:33:30.117425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.116783 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" podUID="5a064383-9f17-459d-b2d6-0600806167fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:33:30.637417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.629525 5824 generic.go:356] "Generic (PLEG): container finished" podID="5a064383-9f17-459d-b2d6-0600806167fd" containerID="ba8418417d19b371a43d712ba6a3ed9bbe06be6d7957e23d12603a0efa0d4ca6" exitCode=0 Oct 30 14:33:30.637417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.637296 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" event={"ID":"5a064383-9f17-459d-b2d6-0600806167fd","Type":"ContainerDied","Data":"ba8418417d19b371a43d712ba6a3ed9bbe06be6d7957e23d12603a0efa0d4ca6"} Oct 30 14:33:30.644910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.644657 5824 generic.go:356] "Generic (PLEG): container finished" podID="0fcdf182-ec74-4302-9fbe-1a9068aa263c" containerID="a8d094220567b5f84043a2bd53cc5a75bcc78c13dc9ff34749e78464dcc2e506" exitCode=0 Oct 30 14:33:30.644910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.644893 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" event={"ID":"0fcdf182-ec74-4302-9fbe-1a9068aa263c","Type":"ContainerDied","Data":"a8d094220567b5f84043a2bd53cc5a75bcc78c13dc9ff34749e78464dcc2e506"} Oct 30 14:33:30.692516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.692482 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:33:30.775013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.774981 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:33:30.787984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.787949 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a064383-9f17-459d-b2d6-0600806167fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a064383-9f17-459d-b2d6-0600806167fd-plugins-config-volume\") pod \"5a064383-9f17-459d-b2d6-0600806167fd\" (UID: \"5a064383-9f17-459d-b2d6-0600806167fd\") " Oct 30 14:33:30.788152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.788038 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a064383-9f17-459d-b2d6-0600806167fd-kube-api-access-v2bmj\" (UniqueName: \"kubernetes.io/projected/5a064383-9f17-459d-b2d6-0600806167fd-kube-api-access-v2bmj\") pod \"5a064383-9f17-459d-b2d6-0600806167fd\" (UID: \"5a064383-9f17-459d-b2d6-0600806167fd\") " Oct 30 14:33:30.788152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.788091 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fcdf182-ec74-4302-9fbe-1a9068aa263c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf182-ec74-4302-9fbe-1a9068aa263c-plugins-config-volume\") pod \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\" (UID: \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\") " Oct 30 14:33:30.788309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.788175 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fcdf182-ec74-4302-9fbe-1a9068aa263c-kube-api-access-wwr7s\" (UniqueName: \"kubernetes.io/projected/0fcdf182-ec74-4302-9fbe-1a9068aa263c-kube-api-access-wwr7s\") pod \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\" (UID: \"0fcdf182-ec74-4302-9fbe-1a9068aa263c\") " Oct 30 14:33:30.788506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.788476 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a064383-9f17-459d-b2d6-0600806167fd-plugins-config-volume" pod "5a064383-9f17-459d-b2d6-0600806167fd" (UID: "5a064383-9f17-459d-b2d6-0600806167fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:30.788643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.788500 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fcdf182-ec74-4302-9fbe-1a9068aa263c-plugins-config-volume" pod "0fcdf182-ec74-4302-9fbe-1a9068aa263c" (UID: "0fcdf182-ec74-4302-9fbe-1a9068aa263c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:30.788951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.788928 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a064383-9f17-459d-b2d6-0600806167fd-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:30.789035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.788959 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fcdf182-ec74-4302-9fbe-1a9068aa263c-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:30.791079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.791017 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a064383-9f17-459d-b2d6-0600806167fd-kube-api-access-v2bmj" pod "5a064383-9f17-459d-b2d6-0600806167fd" (UID: "5a064383-9f17-459d-b2d6-0600806167fd"). InnerVolumeSpecName "kube-api-access-v2bmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:30.797315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.797290 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcdf182-ec74-4302-9fbe-1a9068aa263c-kube-api-access-wwr7s" pod "0fcdf182-ec74-4302-9fbe-1a9068aa263c" (UID: "0fcdf182-ec74-4302-9fbe-1a9068aa263c"). InnerVolumeSpecName "kube-api-access-wwr7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:30.891213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.890354 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bmj\" (UniqueName: \"kubernetes.io/projected/5a064383-9f17-459d-b2d6-0600806167fd-kube-api-access-v2bmj\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:30.891213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:30.890408 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwr7s\" (UniqueName: \"kubernetes.io/projected/0fcdf182-ec74-4302-9fbe-1a9068aa263c-kube-api-access-wwr7s\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:31.653714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.653633 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" Oct 30 14:33:31.653714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.653642 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65" event={"ID":"5a064383-9f17-459d-b2d6-0600806167fd","Type":"ContainerDied","Data":"b621633a13bb21c7b5255c02a4bda6b93957a435be3a429af1a1c35595da3e62"} Oct 30 14:33:31.653714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.653707 5824 scope.go:122] "RemoveContainer" containerID="ba8418417d19b371a43d712ba6a3ed9bbe06be6d7957e23d12603a0efa0d4ca6" Oct 30 14:33:31.655422 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.655403 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" Oct 30 14:33:31.655887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.655447 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n" event={"ID":"0fcdf182-ec74-4302-9fbe-1a9068aa263c","Type":"ContainerDied","Data":"e74361be8e869314cab9435587dc927359f0d960f82b746ec024eadb0ce2275f"} Oct 30 14:33:31.666011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.665986 5824 scope.go:122] "RemoveContainer" containerID="a8d094220567b5f84043a2bd53cc5a75bcc78c13dc9ff34749e78464dcc2e506" Oct 30 14:33:31.715603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.715572 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65"] Oct 30 14:33:31.726823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.726798 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b1c15-5bccfc8c74-b7c65"] Oct 30 14:33:31.758969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.758923 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n"] Oct 30 14:33:31.763314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:31.763291 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b1c15-754948978c-wql6n"] Oct 30 14:33:32.318179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:32.318145 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fcdf182-ec74-4302-9fbe-1a9068aa263c" path="/var/lib/kubelet/pods/0fcdf182-ec74-4302-9fbe-1a9068aa263c/volumes" Oct 30 14:33:32.318640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:32.318624 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a064383-9f17-459d-b2d6-0600806167fd" path="/var/lib/kubelet/pods/5a064383-9f17-459d-b2d6-0600806167fd/volumes" Oct 30 14:33:33.644545 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:33:33.644509 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:33:39.192506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:39.192468 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt"] Oct 30 14:33:39.193047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:39.192726 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" containerID="cri-o://1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897" gracePeriod=30 Oct 30 14:33:41.319901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:41.319857 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl"] Oct 30 14:33:44.488364 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.488337 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:33:44.569298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569259 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-kube-api-access-dnp29\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-kube-api-access-dnp29\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.569495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569323 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-credential-socket\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.569495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569372 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-socket\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.569495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569416 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-token\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-token\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.569495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569481 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-podinfo\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.569743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569539 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-data\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.569743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569612 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istiod-ca-cert\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.569743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569659 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-certs\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.569743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.569691 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-envoy\") pod \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\" (UID: \"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc\") " Oct 30 14:33:44.570711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.570399 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-credential-socket" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.570711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.570428 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-data" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.570914 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.570858 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istiod-ca-cert" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:33:44.571139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.571114 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-certs" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.571655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.571631 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.571795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.571778 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.571914 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.571901 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.572035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.572015 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.572035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.571694 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-socket" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.573100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.573066 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-podinfo" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:33:44.573424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.573397 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-kube-api-access-dnp29" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "kube-api-access-dnp29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:44.573744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.573719 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-envoy" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:33:44.574286 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.574261 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-token" pod "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" (UID: "587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:33:44.673061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.673036 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.673061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.673061 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnp29\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-kube-api-access-dnp29\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.673269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.673071 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.673269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.673079 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.673269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.673089 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:33:44.756514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.756439 5824 generic.go:356] "Generic (PLEG): container finished" podID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerID="1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897" exitCode=0 Oct 30 14:33:44.756514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.756490 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" Oct 30 14:33:44.756514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.756498 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" event={"ID":"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc","Type":"ContainerDied","Data":"1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897"} Oct 30 14:33:44.756707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.756546 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt" event={"ID":"587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc","Type":"ContainerDied","Data":"fe7e4a29bb815051175db257bc6b017c8b90bc8f48195a67bb3c418ea1fe611e"} Oct 30 14:33:44.756707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.756572 5824 scope.go:122] "RemoveContainer" containerID="1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897" Oct 30 14:33:44.768577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.768554 5824 scope.go:122] "RemoveContainer" containerID="1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897" Oct 30 14:33:44.768846 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:33:44.768826 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897\": container with ID starting with 1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897 not found: ID does not exist" containerID="1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897" Oct 30 14:33:44.768917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.768858 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897"} err="failed to get container status \"1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897\": rpc error: code = NotFound desc = could not find container \"1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897\": container with ID starting with 1cd19c22022083ebc135591458d4f9a2308c194dcd2626fb52aa36ff57199897 not found: ID does not exist" Oct 30 14:33:44.818766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.818742 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt"] Oct 30 14:33:44.831241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:44.828275 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0b1c15-openshift-default-7d89c8877c-48jxt"] Oct 30 14:33:46.318679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:46.318648 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" path="/var/lib/kubelet/pods/587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc/volumes" Oct 30 14:33:46.320749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:46.320724 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" podUID="b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" containerName="payload-processing" containerID="cri-o://ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041" gracePeriod=25 Oct 30 14:33:54.872488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:33:54.872447 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" podUID="b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:34:03.642069 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:34:03.642026 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:34:04.872403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:04.872362 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" podUID="b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:34:06.590457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.590401 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:34:06.649549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.649524 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-kube-api-access-w2qwx\" (UniqueName: \"kubernetes.io/projected/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-kube-api-access-w2qwx\") pod \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\" (UID: \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\") " Oct 30 14:34:06.649776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.649758 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-plugins-config-volume\") pod \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\" (UID: \"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9\") " Oct 30 14:34:06.650317 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.650275 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-plugins-config-volume" pod "b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" (UID: "b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:06.651832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.651794 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-kube-api-access-w2qwx" pod "b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" (UID: "b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9"). InnerVolumeSpecName "kube-api-access-w2qwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:06.751534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.751500 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qwx\" (UniqueName: \"kubernetes.io/projected/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-kube-api-access-w2qwx\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:06.751534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.751529 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:06.926907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.926872 5824 generic.go:356] "Generic (PLEG): container finished" podID="b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" containerID="ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041" exitCode=0 Oct 30 14:34:06.927077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.926929 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" Oct 30 14:34:06.927077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.926942 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" event={"ID":"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9","Type":"ContainerDied","Data":"ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041"} Oct 30 14:34:06.927077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.926988 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl" event={"ID":"b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9","Type":"ContainerDied","Data":"0da0956edb18c1026175d5dc83f2f28f9e0e840edbb21b26c2107303e4756678"} Oct 30 14:34:06.927077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.927013 5824 scope.go:122] "RemoveContainer" containerID="ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041" Oct 30 14:34:06.941708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.941689 5824 scope.go:122] "RemoveContainer" containerID="ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041" Oct 30 14:34:06.941988 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:34:06.941966 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041\": container with ID starting with ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041 not found: ID does not exist" containerID="ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041" Oct 30 14:34:06.942033 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.941996 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041"} err="failed to get container status \"ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041\": rpc error: code = NotFound desc = could not find container \"ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041\": container with ID starting with ec325846a94f2f931f2bc6823aff80c33211cba610bcf97ac3a25eddaf13b041 not found: ID does not exist" Oct 30 14:34:06.988876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.988855 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl"] Oct 30 14:34:06.996202 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:06.996171 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-019e5b-6578bd7576-w9dfl"] Oct 30 14:34:08.318078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:08.318047 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" path="/var/lib/kubelet/pods/b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9/volumes" Oct 30 14:34:30.104121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:30.104083 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc"] Oct 30 14:34:30.104539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:30.104365 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" containerID="cri-o://047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1" gracePeriod=30 Oct 30 14:34:33.655957 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:34:33.655883 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:34:35.445777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.445748 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:34:35.540882 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.540841 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-envoy\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.540903 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-data\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.540992 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istiod-ca-cert\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541046 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-kube-api-access-nn7sl\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-kube-api-access-nn7sl\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541210 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-data" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:35.541294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541216 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-podinfo\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541333 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-certs\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541373 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-credential-socket\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541431 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-socket\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541555 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-token\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-token\") pod \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\" (UID: \"f05b8bdf-8389-4448-b6fa-2b995ff94f36\") " Oct 30 14:34:35.541653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541597 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istiod-ca-cert" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:35.541848 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.541825 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-credential-socket" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:35.542125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.542102 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-certs" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:35.542320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.542300 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-socket" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:35.542401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.542309 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:35.542401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.542338 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:35.542401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.542354 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:35.542401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.542372 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:35.546010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.545985 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-podinfo" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:34:35.547369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.547340 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-token" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:35.547369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.547339 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-envoy" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:34:35.547846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.547822 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-kube-api-access-nn7sl" pod "f05b8bdf-8389-4448-b6fa-2b995ff94f36" (UID: "f05b8bdf-8389-4448-b6fa-2b995ff94f36"). InnerVolumeSpecName "kube-api-access-nn7sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:35.643653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.643622 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:35.643653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.643652 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:35.643858 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.643665 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:35.643858 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.643678 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn7sl\" (UniqueName: \"kubernetes.io/projected/f05b8bdf-8389-4448-b6fa-2b995ff94f36-kube-api-access-nn7sl\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:35.643858 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:35.643690 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f05b8bdf-8389-4448-b6fa-2b995ff94f36-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:36.138925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.138891 5824 generic.go:356] "Generic (PLEG): container finished" podID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerID="047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1" exitCode=0 Oct 30 14:34:36.139132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.138950 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" Oct 30 14:34:36.139132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.138958 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" event={"ID":"f05b8bdf-8389-4448-b6fa-2b995ff94f36","Type":"ContainerDied","Data":"047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1"} Oct 30 14:34:36.139132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.139019 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc" event={"ID":"f05b8bdf-8389-4448-b6fa-2b995ff94f36","Type":"ContainerDied","Data":"42deadc0df850193ef0d9896eb244e7d9324c2a9db0fb395f767889a40cb6ad7"} Oct 30 14:34:36.139132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.139046 5824 scope.go:122] "RemoveContainer" containerID="047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1" Oct 30 14:34:36.155014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.154078 5824 scope.go:122] "RemoveContainer" containerID="306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8" Oct 30 14:34:36.170663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.170636 5824 scope.go:122] "RemoveContainer" containerID="047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1" Oct 30 14:34:36.170990 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:34:36.170968 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1\": container with ID starting with 047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1 not found: ID does not exist" containerID="047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1" Oct 30 14:34:36.171048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.171005 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1"} err="failed to get container status \"047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1\": rpc error: code = NotFound desc = could not find container \"047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1\": container with ID starting with 047a832aed2bbffa9ad103d7e66d2b29d3d8cf8e4a5d1c950a1d1256e832bfc1 not found: ID does not exist" Oct 30 14:34:36.171048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.171028 5824 scope.go:122] "RemoveContainer" containerID="306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8" Oct 30 14:34:36.171331 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:34:36.171303 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05b8bdf_8389_4448_b6fa_2b995ff94f36.slice/crio-42deadc0df850193ef0d9896eb244e7d9324c2a9db0fb395f767889a40cb6ad7\": RecentStats: unable to find data in memory cache]" Oct 30 14:34:36.171458 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:34:36.171351 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8\": container with ID starting with 306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8 not found: ID does not exist" containerID="306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8" Oct 30 14:34:36.171458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.171369 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8"} err="failed to get container status \"306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8\": rpc error: code = NotFound desc = could not find container \"306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8\": container with ID starting with 306238eb5100718e024dd485804e9f2c7d147bb09261a385dcd9b87e28208eb8 not found: ID does not exist" Oct 30 14:34:36.203168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.203138 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc"] Oct 30 14:34:36.218019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.217983 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-019e5b-openshift-default-7d7cb97b95-cdrkc"] Oct 30 14:34:36.320982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:36.320947 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" path="/var/lib/kubelet/pods/f05b8bdf-8389-4448-b6fa-2b995ff94f36/volumes" Oct 30 14:34:40.868518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.868483 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78"] Oct 30 14:34:40.869180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869166 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869181 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869203 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" containerName="payload-processing" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869209 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" containerName="payload-processing" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869218 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869223 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869231 5824 cpu_manager.go:414] "containerMap: removing container" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869237 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869248 5824 cpu_manager.go:414] "containerMap: removing container" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" Oct 30 14:34:40.869251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869252 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869271 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6c9c632e-a54c-40ed-8b83-3f86509c9ae4" containerName="revoke-keys" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869276 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9c632e-a54c-40ed-8b83-3f86509c9ae4" containerName="revoke-keys" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869283 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e55fba24-dca9-4b03-b9a0-645ee11072a9" containerName="payload-pre-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869287 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55fba24-dca9-4b03-b9a0-645ee11072a9" containerName="payload-pre-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869295 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869299 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869310 5824 cpu_manager.go:414] "containerMap: removing container" podUID="0fcdf182-ec74-4302-9fbe-1a9068aa263c" containerName="payload-pre-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869315 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcdf182-ec74-4302-9fbe-1a9068aa263c" containerName="payload-pre-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869326 5824 cpu_manager.go:414] "containerMap: removing container" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869331 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869336 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5a064383-9f17-459d-b2d6-0600806167fd" containerName="payload-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869340 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a064383-9f17-459d-b2d6-0600806167fd" containerName="payload-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869356 5824 cpu_manager.go:414] "containerMap: removing container" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869361 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869372 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869376 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869382 5824 cpu_manager.go:414] "containerMap: removing container" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869387 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869398 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6a948cff-ab50-424c-bc53-b5d16ff54c84" containerName="payload-pre-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869403 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a948cff-ab50-424c-bc53-b5d16ff54c84" containerName="payload-pre-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869500 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9c632e-a54c-40ed-8b83-3f86509c9ae4" containerName="revoke-keys" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869507 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55fba24-dca9-4b03-b9a0-645ee11072a9" containerName="payload-pre-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869513 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869520 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcdf182-ec74-4302-9fbe-1a9068aa263c" containerName="payload-pre-processing" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869527 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="587ac0b1-9b6f-47ee-9c1e-f924a0a62ffc" containerName="istio-proxy" Oct 30 14:34:40.869528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869533 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a064383-9f17-459d-b2d6-0600806167fd" containerName="payload-processing" Oct 30 14:34:40.870356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869542 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e049a0-9ed9-49c5-83bf-54f76f39ee44" containerName="istio-proxy" Oct 30 14:34:40.870356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869550 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e05408-2277-4bfd-a618-b8cbd2249911" containerName="istio-proxy" Oct 30 14:34:40.870356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869556 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="da87cb61-ccfa-49c2-9c39-cd6583a00d3c" containerName="istio-proxy" Oct 30 14:34:40.870356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869563 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a948cff-ab50-424c-bc53-b5d16ff54c84" containerName="payload-pre-processing" Oct 30 14:34:40.870356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869569 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bebc04-7fc4-45a4-9468-4fd4d7f7a2d9" containerName="payload-processing" Oct 30 14:34:40.870356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869576 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2c31c7-c84d-4e01-a1db-53b637f7882d" containerName="istio-proxy" Oct 30 14:34:40.870356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869581 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="159f5cf3-e422-4f1a-9769-cb20b007d4b0" containerName="istio-proxy" Oct 30 14:34:40.870356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.869586 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd369177-a300-45ac-a896-65e097ce54ed" containerName="istio-proxy" Oct 30 14:34:40.874649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.874625 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:40.892520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.892488 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78"] Oct 30 14:34:40.900496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.900463 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:34:40.900633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:40.900551 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:34:41.019761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.019681 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.019761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.019744 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqlft\" (UniqueName: \"kubernetes.io/projected/8a109044-de80-4edf-b7e8-cc82422eb9d8-kube-api-access-jqlft\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.020005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.019942 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.120984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.120892 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqlft\" (UniqueName: \"kubernetes.io/projected/8a109044-de80-4edf-b7e8-cc82422eb9d8-kube-api-access-jqlft\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.121158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.121041 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.121263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.121245 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.121767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.121742 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.123553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.123532 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.148227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.148162 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqlft\" (UniqueName: \"kubernetes.io/projected/8a109044-de80-4edf-b7e8-cc82422eb9d8-kube-api-access-jqlft\") pod \"maas-api-revoke-sub-4059a2d7-ghz78\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.162273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.157641 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq"] Oct 30 14:34:41.194503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.194481 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:41.513503 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:34:41.513466 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a109044_de80_4edf_b7e8_cc82422eb9d8.slice/crio-19eed2d062b0f64a36c5b80aff81a6f7fc5c924303bdcc1e01971a5e3b337b47 WatchSource:0}: Error finding container 19eed2d062b0f64a36c5b80aff81a6f7fc5c924303bdcc1e01971a5e3b337b47: Status 404 returned error can't find the container with id 19eed2d062b0f64a36c5b80aff81a6f7fc5c924303bdcc1e01971a5e3b337b47 Oct 30 14:34:41.514031 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:41.514000 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78"] Oct 30 14:34:42.184613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:42.184576 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" event={"ID":"8a109044-de80-4edf-b7e8-cc82422eb9d8","Type":"ContainerStarted","Data":"14d45ad4823fac09c75c0590651f1fc6883c3f75d210fec6bfdf744b5449ff45"} Oct 30 14:34:42.185048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:42.184656 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" event={"ID":"8a109044-de80-4edf-b7e8-cc82422eb9d8","Type":"ContainerStarted","Data":"19eed2d062b0f64a36c5b80aff81a6f7fc5c924303bdcc1e01971a5e3b337b47"} Oct 30 14:34:42.231338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:42.231286 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" podStartSLOduration=2.231270381 podStartE2EDuration="2.231270381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:34:42.230481302 +0000 UTC m=+2712.030986747" watchObservedRunningTime="2026-10-30 14:34:42.231270381 +0000 UTC m=+2712.031775824" Oct 30 14:34:43.192434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:43.192398 5824 generic.go:356] "Generic (PLEG): container finished" podID="8a109044-de80-4edf-b7e8-cc82422eb9d8" containerID="14d45ad4823fac09c75c0590651f1fc6883c3f75d210fec6bfdf744b5449ff45" exitCode=0 Oct 30 14:34:43.192823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:43.192445 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" event={"ID":"8a109044-de80-4edf-b7e8-cc82422eb9d8","Type":"ContainerDied","Data":"14d45ad4823fac09c75c0590651f1fc6883c3f75d210fec6bfdf744b5449ff45"} Oct 30 14:34:44.354983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.354958 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:44.470283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.470164 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a109044-de80-4edf-b7e8-cc82422eb9d8-kube-api-access-jqlft\" (UniqueName: \"kubernetes.io/projected/8a109044-de80-4edf-b7e8-cc82422eb9d8-kube-api-access-jqlft\") pod \"8a109044-de80-4edf-b7e8-cc82422eb9d8\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " Oct 30 14:34:44.470471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.470327 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-openshift-ca\") pod \"8a109044-de80-4edf-b7e8-cc82422eb9d8\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " Oct 30 14:34:44.470471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.470437 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-xks-ca\") pod \"8a109044-de80-4edf-b7e8-cc82422eb9d8\" (UID: \"8a109044-de80-4edf-b7e8-cc82422eb9d8\") " Oct 30 14:34:44.470740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.470710 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-openshift-ca" pod "8a109044-de80-4edf-b7e8-cc82422eb9d8" (UID: "8a109044-de80-4edf-b7e8-cc82422eb9d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:44.471384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.471350 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:44.472707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.472680 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-xks-ca" pod "8a109044-de80-4edf-b7e8-cc82422eb9d8" (UID: "8a109044-de80-4edf-b7e8-cc82422eb9d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:34:44.472827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.472792 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a109044-de80-4edf-b7e8-cc82422eb9d8-kube-api-access-jqlft" pod "8a109044-de80-4edf-b7e8-cc82422eb9d8" (UID: "8a109044-de80-4edf-b7e8-cc82422eb9d8"). InnerVolumeSpecName "kube-api-access-jqlft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:44.572956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.572922 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a109044-de80-4edf-b7e8-cc82422eb9d8-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:44.572956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:44.572952 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqlft\" (UniqueName: \"kubernetes.io/projected/8a109044-de80-4edf-b7e8-cc82422eb9d8-kube-api-access-jqlft\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:45.207570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:45.207538 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" Oct 30 14:34:45.207763 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:45.207562 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78" event={"ID":"8a109044-de80-4edf-b7e8-cc82422eb9d8","Type":"ContainerDied","Data":"19eed2d062b0f64a36c5b80aff81a6f7fc5c924303bdcc1e01971a5e3b337b47"} Oct 30 14:34:45.207763 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:45.207617 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19eed2d062b0f64a36c5b80aff81a6f7fc5c924303bdcc1e01971a5e3b337b47" Oct 30 14:34:46.158433 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:46.158375 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" podUID="98701cd7-fc28-4901-8076-1bd0e8c6f26c" containerName="payload-processing" containerID="cri-o://310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f" gracePeriod=25 Oct 30 14:34:46.448077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:46.448043 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:34:46.599027 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:46.598998 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98701cd7-fc28-4901-8076-1bd0e8c6f26c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98701cd7-fc28-4901-8076-1bd0e8c6f26c-plugins-config-volume\") pod \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\" (UID: \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\") " Oct 30 14:34:46.599185 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:46.599056 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98701cd7-fc28-4901-8076-1bd0e8c6f26c-kube-api-access-2zw9b\" (UniqueName: \"kubernetes.io/projected/98701cd7-fc28-4901-8076-1bd0e8c6f26c-kube-api-access-2zw9b\") pod \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\" (UID: \"98701cd7-fc28-4901-8076-1bd0e8c6f26c\") " Oct 30 14:34:46.599427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:46.599401 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98701cd7-fc28-4901-8076-1bd0e8c6f26c-plugins-config-volume" pod "98701cd7-fc28-4901-8076-1bd0e8c6f26c" (UID: "98701cd7-fc28-4901-8076-1bd0e8c6f26c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:46.601103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:46.601070 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98701cd7-fc28-4901-8076-1bd0e8c6f26c-kube-api-access-2zw9b" pod "98701cd7-fc28-4901-8076-1bd0e8c6f26c" (UID: "98701cd7-fc28-4901-8076-1bd0e8c6f26c"). InnerVolumeSpecName "kube-api-access-2zw9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:46.700808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:46.700783 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98701cd7-fc28-4901-8076-1bd0e8c6f26c-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:46.700808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:46.700807 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zw9b\" (UniqueName: \"kubernetes.io/projected/98701cd7-fc28-4901-8076-1bd0e8c6f26c-kube-api-access-2zw9b\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:47.223523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.223492 5824 generic.go:356] "Generic (PLEG): container finished" podID="98701cd7-fc28-4901-8076-1bd0e8c6f26c" containerID="310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f" exitCode=0 Oct 30 14:34:47.223922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.223538 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" Oct 30 14:34:47.223922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.223578 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" event={"ID":"98701cd7-fc28-4901-8076-1bd0e8c6f26c","Type":"ContainerDied","Data":"310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f"} Oct 30 14:34:47.223922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.223632 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq" event={"ID":"98701cd7-fc28-4901-8076-1bd0e8c6f26c","Type":"ContainerDied","Data":"d02410411baf6fadf99e19f7c2726013a6dbb46d3e90c16faff1b0b0b8f440a4"} Oct 30 14:34:47.223922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.223656 5824 scope.go:122] "RemoveContainer" containerID="310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f" Oct 30 14:34:47.235318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.235298 5824 scope.go:122] "RemoveContainer" containerID="310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f" Oct 30 14:34:47.235605 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:34:47.235586 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f\": container with ID starting with 310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f not found: ID does not exist" containerID="310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f" Oct 30 14:34:47.235679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.235611 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f"} err="failed to get container status \"310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f\": rpc error: code = NotFound desc = could not find container \"310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f\": container with ID starting with 310a42ce8d6954dac4c081d86fddaeecde091e3b94630b4464c5c4e90937cc3f not found: ID does not exist" Oct 30 14:34:47.317123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.317100 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq"] Oct 30 14:34:47.323764 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:47.323745 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d3696-b6fbdfb88-wdqxq"] Oct 30 14:34:48.318488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:48.318456 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98701cd7-fc28-4901-8076-1bd0e8c6f26c" path="/var/lib/kubelet/pods/98701cd7-fc28-4901-8076-1bd0e8c6f26c/volumes" Oct 30 14:34:51.206822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.206789 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2"] Oct 30 14:34:51.207579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.207561 5824 cpu_manager.go:414] "containerMap: removing container" podUID="8a109044-de80-4edf-b7e8-cc82422eb9d8" containerName="revoke-keys" Oct 30 14:34:51.207651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.207581 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a109044-de80-4edf-b7e8-cc82422eb9d8" containerName="revoke-keys" Oct 30 14:34:51.207651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.207618 5824 cpu_manager.go:414] "containerMap: removing container" podUID="98701cd7-fc28-4901-8076-1bd0e8c6f26c" containerName="payload-processing" Oct 30 14:34:51.207651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.207623 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98701cd7-fc28-4901-8076-1bd0e8c6f26c" containerName="payload-processing" Oct 30 14:34:51.207749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.207729 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a109044-de80-4edf-b7e8-cc82422eb9d8" containerName="revoke-keys" Oct 30 14:34:51.207749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.207740 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05b8bdf-8389-4448-b6fa-2b995ff94f36" containerName="istio-proxy" Oct 30 14:34:51.207813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.207751 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="98701cd7-fc28-4901-8076-1bd0e8c6f26c" containerName="payload-processing" Oct 30 14:34:51.211439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.211423 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.228777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.228756 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2"] Oct 30 14:34:51.238413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.238389 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:34:51.238413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.238401 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:34:51.257331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.257298 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbjb\" (UniqueName: \"kubernetes.io/projected/2022b464-958d-4738-8a24-5661fa6646f8-kube-api-access-5bbjb\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.257500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.257477 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.257736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.257716 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.358935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.358900 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.359069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.358964 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbjb\" (UniqueName: \"kubernetes.io/projected/2022b464-958d-4738-8a24-5661fa6646f8-kube-api-access-5bbjb\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.359149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.359063 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.359593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.359573 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.361376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.361359 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.385390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.385364 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbjb\" (UniqueName: \"kubernetes.io/projected/2022b464-958d-4738-8a24-5661fa6646f8-kube-api-access-5bbjb\") pod \"maas-api-revoke-sub-ed2673e5-gvpq2\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.526288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.526224 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:51.830661 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:34:51.830621 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2022b464_958d_4738_8a24_5661fa6646f8.slice/crio-021d369bf166f8cd5116e822eb2abdce57618cd1d97088dbd5344c3333ff6b03 WatchSource:0}: Error finding container 021d369bf166f8cd5116e822eb2abdce57618cd1d97088dbd5344c3333ff6b03: Status 404 returned error can't find the container with id 021d369bf166f8cd5116e822eb2abdce57618cd1d97088dbd5344c3333ff6b03 Oct 30 14:34:51.830952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:51.830932 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2"] Oct 30 14:34:52.262495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:52.262458 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" event={"ID":"2022b464-958d-4738-8a24-5661fa6646f8","Type":"ContainerStarted","Data":"3629632fd2861982e5c09106bf64439e60d1d57224d4a60b8efd9ee9c23fbe2d"} Oct 30 14:34:52.262864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:52.262515 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" event={"ID":"2022b464-958d-4738-8a24-5661fa6646f8","Type":"ContainerStarted","Data":"021d369bf166f8cd5116e822eb2abdce57618cd1d97088dbd5344c3333ff6b03"} Oct 30 14:34:53.271283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:53.271249 5824 generic.go:356] "Generic (PLEG): container finished" podID="2022b464-958d-4738-8a24-5661fa6646f8" containerID="3629632fd2861982e5c09106bf64439e60d1d57224d4a60b8efd9ee9c23fbe2d" exitCode=0 Oct 30 14:34:53.271674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:53.271336 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" event={"ID":"2022b464-958d-4738-8a24-5661fa6646f8","Type":"ContainerDied","Data":"3629632fd2861982e5c09106bf64439e60d1d57224d4a60b8efd9ee9c23fbe2d"} Oct 30 14:34:54.434306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.434281 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:34:54.499755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.499725 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-openshift-ca\") pod \"2022b464-958d-4738-8a24-5661fa6646f8\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " Oct 30 14:34:54.499937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.499822 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2022b464-958d-4738-8a24-5661fa6646f8-kube-api-access-5bbjb\" (UniqueName: \"kubernetes.io/projected/2022b464-958d-4738-8a24-5661fa6646f8-kube-api-access-5bbjb\") pod \"2022b464-958d-4738-8a24-5661fa6646f8\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " Oct 30 14:34:54.499937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.499916 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-xks-ca\") pod \"2022b464-958d-4738-8a24-5661fa6646f8\" (UID: \"2022b464-958d-4738-8a24-5661fa6646f8\") " Oct 30 14:34:54.500163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.500130 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-openshift-ca" pod "2022b464-958d-4738-8a24-5661fa6646f8" (UID: "2022b464-958d-4738-8a24-5661fa6646f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:34:54.500606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.500582 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:54.502156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.502134 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-xks-ca" pod "2022b464-958d-4738-8a24-5661fa6646f8" (UID: "2022b464-958d-4738-8a24-5661fa6646f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:34:54.502566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.502541 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2022b464-958d-4738-8a24-5661fa6646f8-kube-api-access-5bbjb" pod "2022b464-958d-4738-8a24-5661fa6646f8" (UID: "2022b464-958d-4738-8a24-5661fa6646f8"). InnerVolumeSpecName "kube-api-access-5bbjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:34:54.601563 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.601496 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbjb\" (UniqueName: \"kubernetes.io/projected/2022b464-958d-4738-8a24-5661fa6646f8-kube-api-access-5bbjb\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:54.601563 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:54.601521 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2022b464-958d-4738-8a24-5661fa6646f8-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:34:55.286102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:55.286067 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" event={"ID":"2022b464-958d-4738-8a24-5661fa6646f8","Type":"ContainerDied","Data":"021d369bf166f8cd5116e822eb2abdce57618cd1d97088dbd5344c3333ff6b03"} Oct 30 14:34:55.286288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:55.286125 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="021d369bf166f8cd5116e822eb2abdce57618cd1d97088dbd5344c3333ff6b03" Oct 30 14:34:55.286288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:34:55.286079 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2" Oct 30 14:35:03.644179 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:35:03.644129 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:35:14.272712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.272678 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl"] Oct 30 14:35:14.273394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.273379 5824 cpu_manager.go:414] "containerMap: removing container" podUID="2022b464-958d-4738-8a24-5661fa6646f8" containerName="revoke-keys" Oct 30 14:35:14.273394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.273395 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2022b464-958d-4738-8a24-5661fa6646f8" containerName="revoke-keys" Oct 30 14:35:14.273536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.273526 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="2022b464-958d-4738-8a24-5661fa6646f8" containerName="revoke-keys" Oct 30 14:35:14.277146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.277127 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:14.296137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.296109 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl"] Oct 30 14:35:14.305679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.305660 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-f1f206\"" Oct 30 14:35:14.371115 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.371072 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtr6\" (UniqueName: \"kubernetes.io/projected/7d4d4407-499b-42c4-8a84-7a30c4b2f608-kube-api-access-2dtr6\") pod \"payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl\" (UID: \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:14.371387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.371362 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4d4407-499b-42c4-8a84-7a30c4b2f608-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl\" (UID: \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:14.476316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.476276 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4d4407-499b-42c4-8a84-7a30c4b2f608-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl\" (UID: \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:14.476763 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.476735 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtr6\" (UniqueName: \"kubernetes.io/projected/7d4d4407-499b-42c4-8a84-7a30c4b2f608-kube-api-access-2dtr6\") pod \"payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl\" (UID: \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:14.478461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.477901 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4d4407-499b-42c4-8a84-7a30c4b2f608-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl\" (UID: \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:14.509175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.509142 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtr6\" (UniqueName: \"kubernetes.io/projected/7d4d4407-499b-42c4-8a84-7a30c4b2f608-kube-api-access-2dtr6\") pod \"payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl\" (UID: \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:14.592950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.592859 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:14.898393 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:35:14.898364 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4d4407_499b_42c4_8a84_7a30c4b2f608.slice/crio-fcf016bccffdf24a3d426d6b5619015150f8ca976ff4899bf4caff128d0c4e94 WatchSource:0}: Error finding container fcf016bccffdf24a3d426d6b5619015150f8ca976ff4899bf4caff128d0c4e94: Status 404 returned error can't find the container with id fcf016bccffdf24a3d426d6b5619015150f8ca976ff4899bf4caff128d0c4e94 Oct 30 14:35:14.899063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:14.899034 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl"] Oct 30 14:35:15.439173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:15.439136 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" event={"ID":"7d4d4407-499b-42c4-8a84-7a30c4b2f608","Type":"ContainerStarted","Data":"d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76"} Oct 30 14:35:15.439584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:15.439218 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" event={"ID":"7d4d4407-499b-42c4-8a84-7a30c4b2f608","Type":"ContainerStarted","Data":"fcf016bccffdf24a3d426d6b5619015150f8ca976ff4899bf4caff128d0c4e94"} Oct 30 14:35:15.439584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:15.439357 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:15.487356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:15.487297 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" podStartSLOduration=1.48728437 podStartE2EDuration="1.48728437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:35:15.486580755 +0000 UTC m=+2745.287086213" watchObservedRunningTime="2026-10-30 14:35:15.48728437 +0000 UTC m=+2745.287789814" Oct 30 14:35:26.449722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:35:26.449683 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:35:33.651739 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:35:33.651697 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:36:00.039298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:36:00.039262 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks"] Oct 30 14:36:00.052352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:36:00.052320 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dcf5e6-95zks"] Oct 30 14:36:00.318439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:36:00.318364 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b20ce00-1dca-4657-8a61-2bf8e2cffe09" path="/var/lib/kubelet/pods/1b20ce00-1dca-4657-8a61-2bf8e2cffe09/volumes" Oct 30 14:36:03.644315 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:36:03.644280 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:36:12.035845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:36:12.035771 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph"] Oct 30 14:36:12.049032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:36:12.049004 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db98211-dhhph"] Oct 30 14:36:12.319531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:36:12.319453 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a56a75-bf25-4803-a2c0-0b24abb0ff94" path="/var/lib/kubelet/pods/72a56a75-bf25-4803-a2c0-0b24abb0ff94/volumes" Oct 30 14:36:33.641505 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:36:33.641464 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:37:03.642295 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:37:03.642257 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:37:22.491302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.491268 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn"] Oct 30 14:37:22.495475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.495452 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:22.515129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.515097 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn"] Oct 30 14:37:22.525034 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.525009 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e50e63\"" Oct 30 14:37:22.525034 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.525019 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e50e63-dockercfg-4dtx5\"" Oct 30 14:37:22.586902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.586867 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6"] Oct 30 14:37:22.592041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.591992 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:22.623804 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.623758 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6"] Oct 30 14:37:22.624982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.624950 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17d5a07-9932-42b2-b544-43bb97373bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn\" (UID: \"e17d5a07-9932-42b2-b544-43bb97373bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:22.625129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.625090 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfh9\" (UniqueName: \"kubernetes.io/projected/e17d5a07-9932-42b2-b544-43bb97373bae-kube-api-access-twfh9\") pod \"payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn\" (UID: \"e17d5a07-9932-42b2-b544-43bb97373bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:22.725939 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.725902 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfh9\" (UniqueName: \"kubernetes.io/projected/e17d5a07-9932-42b2-b544-43bb97373bae-kube-api-access-twfh9\") pod \"payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn\" (UID: \"e17d5a07-9932-42b2-b544-43bb97373bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:22.726149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.725960 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6\" (UID: \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:22.726149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.726108 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7qj\" (UniqueName: \"kubernetes.io/projected/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-kube-api-access-kz7qj\") pod \"payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6\" (UID: \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:22.726327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.726185 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17d5a07-9932-42b2-b544-43bb97373bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn\" (UID: \"e17d5a07-9932-42b2-b544-43bb97373bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:22.726919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.726899 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17d5a07-9932-42b2-b544-43bb97373bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn\" (UID: \"e17d5a07-9932-42b2-b544-43bb97373bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:22.752296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.752219 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfh9\" (UniqueName: \"kubernetes.io/projected/e17d5a07-9932-42b2-b544-43bb97373bae-kube-api-access-twfh9\") pod \"payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn\" (UID: \"e17d5a07-9932-42b2-b544-43bb97373bae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:22.809594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.809561 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:22.827566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.827536 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7qj\" (UniqueName: \"kubernetes.io/projected/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-kube-api-access-kz7qj\") pod \"payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6\" (UID: \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:22.827704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.827688 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6\" (UID: \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:22.828363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.828341 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6\" (UID: \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:22.853832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.853803 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7qj\" (UniqueName: \"kubernetes.io/projected/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-kube-api-access-kz7qj\") pod \"payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6\" (UID: \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:22.918913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:22.918234 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:23.129319 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:37:23.129285 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17d5a07_9932_42b2_b544_43bb97373bae.slice/crio-918e9868fc62b42f010ced270394bd7c80b2f8dd58054947d77fe9438e3f78ae WatchSource:0}: Error finding container 918e9868fc62b42f010ced270394bd7c80b2f8dd58054947d77fe9438e3f78ae: Status 404 returned error can't find the container with id 918e9868fc62b42f010ced270394bd7c80b2f8dd58054947d77fe9438e3f78ae Oct 30 14:37:23.129602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.129579 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn"] Oct 30 14:37:23.240874 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:37:23.240848 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb833a15f_4b7c_4c73_93c5_cd133c8c2b2d.slice/crio-9bd6776820aa8f3c26cb7b7625dbc193c8d955a320084d15202019ed0b542c96 WatchSource:0}: Error finding container 9bd6776820aa8f3c26cb7b7625dbc193c8d955a320084d15202019ed0b542c96: Status 404 returned error can't find the container with id 9bd6776820aa8f3c26cb7b7625dbc193c8d955a320084d15202019ed0b542c96 Oct 30 14:37:23.241350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.241326 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6"] Oct 30 14:37:23.407448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.407369 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4"] Oct 30 14:37:23.411818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.411802 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.434158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.434132 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4"] Oct 30 14:37:23.442767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.442738 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6e91df-openshift-default-dockercfg-plsv2\"" Oct 30 14:37:23.457564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.457529 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" event={"ID":"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d","Type":"ContainerStarted","Data":"4ed4391997996e7bf0c7628e87ce1182a838fc42d81ad59b74706cfc6bf7aaac"} Oct 30 14:37:23.457730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.457595 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" event={"ID":"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d","Type":"ContainerStarted","Data":"9bd6776820aa8f3c26cb7b7625dbc193c8d955a320084d15202019ed0b542c96"} Oct 30 14:37:23.458253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.458226 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:23.460941 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.460913 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" event={"ID":"e17d5a07-9932-42b2-b544-43bb97373bae","Type":"ContainerStarted","Data":"a2821e2da519a57979fd0aee314ffba06171c4930202a6d7e78a281b3d3b0179"} Oct 30 14:37:23.461080 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.460963 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" event={"ID":"e17d5a07-9932-42b2-b544-43bb97373bae","Type":"ContainerStarted","Data":"918e9868fc62b42f010ced270394bd7c80b2f8dd58054947d77fe9438e3f78ae"} Oct 30 14:37:23.465284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.465260 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:37:23.514218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.514099 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" podStartSLOduration=1.5140846049999999 podStartE2EDuration="1.514084605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:23.513734642 +0000 UTC m=+2873.314240086" watchObservedRunningTime="2026-10-30 14:37:23.514084605 +0000 UTC m=+2873.314590056" Oct 30 14:37:23.540039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540014 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.540162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540054 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-envoy\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.540162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540076 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-data\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.540307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540221 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-podinfo\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.540307 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540259 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhhv\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-kube-api-access-5jhhv\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.540444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540428 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-certs\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.540498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540465 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-credential-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.540498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540490 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce1b7ad-c03a-457d-9f05-100e28d5886e-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.540628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.540599 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-token\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.558610 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.558568 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" podStartSLOduration=1.558556353 podStartE2EDuration="1.558556353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:23.557354098 +0000 UTC m=+2873.357859566" watchObservedRunningTime="2026-10-30 14:37:23.558556353 +0000 UTC m=+2873.359061832" Oct 30 14:37:23.642200 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642159 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-token\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642227 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642249 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-envoy\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642266 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-data\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642310 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-podinfo\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642336 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhhv\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-kube-api-access-5jhhv\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642515 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-certs\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642570 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-credential-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642611 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce1b7ad-c03a-457d-9f05-100e28d5886e-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642641 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.642942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.642918 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-certs\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.643085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.643062 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-credential-socket\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.643164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.643119 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-data\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.643303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.643284 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce1b7ad-c03a-457d-9f05-100e28d5886e-istiod-ca-cert\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.645160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.645138 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-envoy\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.645397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.645380 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-podinfo\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.668336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.668289 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-token\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.668881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.668810 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhhv\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-kube-api-access-5jhhv\") pod \"e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:23.725990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:23.725958 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:24.055834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:24.055805 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4"] Oct 30 14:37:24.056359 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:37:24.056320 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce1b7ad_c03a_457d_9f05_100e28d5886e.slice/crio-a8fc84f82186ee4e5302e507e6f320b9058672e1072ff10e3a36365c43a0482b WatchSource:0}: Error finding container a8fc84f82186ee4e5302e507e6f320b9058672e1072ff10e3a36365c43a0482b: Status 404 returned error can't find the container with id a8fc84f82186ee4e5302e507e6f320b9058672e1072ff10e3a36365c43a0482b Oct 30 14:37:24.059567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:24.059531 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:24.059651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:24.059594 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:24.059651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:24.059626 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:24.471255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:24.471220 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" event={"ID":"bce1b7ad-c03a-457d-9f05-100e28d5886e","Type":"ContainerStarted","Data":"7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76"} Oct 30 14:37:24.471460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:24.471278 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" event={"ID":"bce1b7ad-c03a-457d-9f05-100e28d5886e","Type":"ContainerStarted","Data":"a8fc84f82186ee4e5302e507e6f320b9058672e1072ff10e3a36365c43a0482b"} Oct 30 14:37:24.522898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:24.522667 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" podStartSLOduration=1.5226467270000001 podStartE2EDuration="1.522646727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:24.52144399 +0000 UTC m=+2874.321949435" watchObservedRunningTime="2026-10-30 14:37:24.522646727 +0000 UTC m=+2874.323152171" Oct 30 14:37:24.727016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:24.726923 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:25.727956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:25.727915 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 14:37:25.728399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:25.727975 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 14:37:26.727021 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:26.726987 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 14:37:26.727247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:26.727043 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 14:37:27.726922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:27.726887 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 14:37:27.727342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:27.726942 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 14:37:27.982263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:27.982175 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp"] Oct 30 14:37:27.986859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:27.986838 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.009111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009072 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.009423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009397 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.009532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009443 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb1faa-2773-4fde-982d-98e73d5dec84-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.009595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009536 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.009595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009583 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-kube-api-access-4nd8p\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.009814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009636 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.009814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009781 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.010045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009870 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.010045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.009901 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.029170 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.029136 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a59ecb-openshift-default-dockercfg-q4p9h\"" Oct 30 14:37:28.038889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.038854 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp"] Oct 30 14:37:28.111626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.111591 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.111772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.111646 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.111772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.111743 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.111872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.111787 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.111872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.111816 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.111989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.111942 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb1faa-2773-4fde-982d-98e73d5dec84-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.112069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.112052 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.112127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.112099 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-kube-api-access-4nd8p\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.112223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.112145 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.112223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.112166 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.112350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.112144 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-credential-socket\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.112419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.112368 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-certs\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.112586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.112566 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-data\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.112776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.112755 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb1faa-2773-4fde-982d-98e73d5dec84-istiod-ca-cert\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.114329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.114309 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-envoy\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.114466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.114445 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-podinfo\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.137464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.137436 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-token\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.137825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.137802 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-kube-api-access-4nd8p\") pod \"e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.301676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.301586 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:28.672478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.672398 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp"] Oct 30 14:37:28.674574 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:37:28.674431 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fb1faa_2773_4fde_982d_98e73d5dec84.slice/crio-21bc713bfdfa57569c910801cd4d5af62278e479ad0a78fb4c32480e536c235b WatchSource:0}: Error finding container 21bc713bfdfa57569c910801cd4d5af62278e479ad0a78fb4c32480e536c235b: Status 404 returned error can't find the container with id 21bc713bfdfa57569c910801cd4d5af62278e479ad0a78fb4c32480e536c235b Oct 30 14:37:28.677406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.677373 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:28.677487 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.677442 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:28.677487 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.677471 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:37:28.727482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.727451 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 14:37:28.728019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:28.727987 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 14:37:29.512231 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:29.511911 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" event={"ID":"69fb1faa-2773-4fde-982d-98e73d5dec84","Type":"ContainerStarted","Data":"044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e"} Oct 30 14:37:29.512231 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:29.511966 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" event={"ID":"69fb1faa-2773-4fde-982d-98e73d5dec84","Type":"ContainerStarted","Data":"21bc713bfdfa57569c910801cd4d5af62278e479ad0a78fb4c32480e536c235b"} Oct 30 14:37:29.596218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:29.595865 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" podStartSLOduration=2.595839475 podStartE2EDuration="2.595839475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:37:29.585710653 +0000 UTC m=+2879.386216100" watchObservedRunningTime="2026-10-30 14:37:29.595839475 +0000 UTC m=+2879.396344922" Oct 30 14:37:29.731217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:29.728608 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 14:37:29.731217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:29.728691 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 14:37:30.301841 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:30.301785 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:30.303645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:30.303617 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 14:37:30.303761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:30.303664 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 14:37:30.727123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:30.727072 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 14:37:30.727331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:30.727153 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 14:37:31.302012 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:31.301980 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 14:37:31.302426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:31.302031 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 14:37:31.732435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:31.732398 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:31.732873 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:31.732851 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:31.733681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:31.733655 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:37:31.933998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:31.933967 5824 scope.go:122] "RemoveContainer" containerID="00372229bc1fbe1992a542df954a42e6d837eac0f94d29619e23c6fbcbe6e298" Oct 30 14:37:31.970064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:31.970026 5824 scope.go:122] "RemoveContainer" containerID="e12c1112da623dae67c006c5728f7ddfe5d9dc331e3f0f490a44795f6e8948c6" Oct 30 14:37:31.981545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:31.981521 5824 scope.go:122] "RemoveContainer" containerID="b07fa1352fa03665596164841ae943f0bed6b1f2ec23f6deb142de687c6e4cea" Oct 30 14:37:32.302387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:32.302299 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 14:37:32.302387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:32.302372 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 14:37:33.302010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:33.301970 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 14:37:33.302228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:33.302027 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 14:37:33.679378 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:37:33.679211 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:37:34.302415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:34.302378 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 14:37:34.302587 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:34.302434 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 14:37:35.454120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:35.454082 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:35.454616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:35.454520 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:35.455826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:35.455801 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:37:35.484515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:35.484484 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:37:35.484657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:37:35.484595 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:38:03.642772 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:38:03.642732 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:38:17.472111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.472075 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft"] Oct 30 14:38:17.472604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.472300 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" podUID="ddd63488-9f69-4048-ae51-e8c9d1df74cb" containerName="discovery" containerID="cri-o://5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8" gracePeriod=30 Oct 30 14:38:17.814512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.814478 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:38:17.911310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.911276 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-kube-api-access-x69ct\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-kube-api-access-x69ct\") pod \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " Oct 30 14:38:17.911499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.911390 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-cacerts\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-cacerts\") pod \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " Oct 30 14:38:17.911499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.911449 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-kubeconfig\") pod \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " Oct 30 14:38:17.911499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.911488 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-token\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-token\") pod \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " Oct 30 14:38:17.911633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.911609 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd63488-9f69-4048-ae51-e8c9d1df74cb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd63488-9f69-4048-ae51-e8c9d1df74cb-local-certs\") pod \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " Oct 30 14:38:17.911727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.911709 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-dns-cert\") pod \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " Oct 30 14:38:17.911837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.911788 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-ca-configmap\") pod \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\" (UID: \"ddd63488-9f69-4048-ae51-e8c9d1df74cb\") " Oct 30 14:38:17.912807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.912774 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-ca-configmap" pod "ddd63488-9f69-4048-ae51-e8c9d1df74cb" (UID: "ddd63488-9f69-4048-ae51-e8c9d1df74cb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:17.914724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.914692 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd63488-9f69-4048-ae51-e8c9d1df74cb-local-certs" pod "ddd63488-9f69-4048-ae51-e8c9d1df74cb" (UID: "ddd63488-9f69-4048-ae51-e8c9d1df74cb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:17.914828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.914782 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-kube-api-access-x69ct" pod "ddd63488-9f69-4048-ae51-e8c9d1df74cb" (UID: "ddd63488-9f69-4048-ae51-e8c9d1df74cb"). InnerVolumeSpecName "kube-api-access-x69ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:17.915055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.915033 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-cacerts" pod "ddd63488-9f69-4048-ae51-e8c9d1df74cb" (UID: "ddd63488-9f69-4048-ae51-e8c9d1df74cb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:38:17.928246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.915209 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-dns-cert" pod "ddd63488-9f69-4048-ae51-e8c9d1df74cb" (UID: "ddd63488-9f69-4048-ae51-e8c9d1df74cb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:38:17.928246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.915262 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-kubeconfig" pod "ddd63488-9f69-4048-ae51-e8c9d1df74cb" (UID: "ddd63488-9f69-4048-ae51-e8c9d1df74cb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:38:17.928246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.915874 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-token" pod "ddd63488-9f69-4048-ae51-e8c9d1df74cb" (UID: "ddd63488-9f69-4048-ae51-e8c9d1df74cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:17.956757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.956718 5824 generic.go:356] "Generic (PLEG): container finished" podID="ddd63488-9f69-4048-ae51-e8c9d1df74cb" containerID="5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8" exitCode=0 Oct 30 14:38:17.956929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.956768 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" Oct 30 14:38:17.956929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.956802 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" event={"ID":"ddd63488-9f69-4048-ae51-e8c9d1df74cb","Type":"ContainerDied","Data":"5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8"} Oct 30 14:38:17.956929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.956866 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft" event={"ID":"ddd63488-9f69-4048-ae51-e8c9d1df74cb","Type":"ContainerDied","Data":"2b506c1011d921f5bb2398da04fa461ced0ed14ca533a3a8fa52f02f011e0a4e"} Oct 30 14:38:17.956929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.956894 5824 scope.go:122] "RemoveContainer" containerID="5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8" Oct 30 14:38:17.973746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.973719 5824 scope.go:122] "RemoveContainer" containerID="5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8" Oct 30 14:38:17.974115 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:38:17.974085 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8\": container with ID starting with 5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8 not found: ID does not exist" containerID="5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8" Oct 30 14:38:17.974267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:17.974119 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8"} err="failed to get container status \"5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8\": rpc error: code = NotFound desc = could not find container \"5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8\": container with ID starting with 5f9e652c12f0e91158bdde7a4439215842cf79ea4c9cc8cb5afd61c5b87587e8 not found: ID does not exist" Oct 30 14:38:18.014001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.013969 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x69ct\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-kube-api-access-x69ct\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:18.014001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.013993 5824 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-cacerts\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:18.014001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.014004 5824 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-kubeconfig\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:18.014246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.014013 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:18.014246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.014023 5824 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd63488-9f69-4048-ae51-e8c9d1df74cb-local-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:18.014246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.014032 5824 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-dns-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:18.014246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.014041 5824 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd63488-9f69-4048-ae51-e8c9d1df74cb-istio-csr-ca-configmap\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:18.016952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.016927 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft"] Oct 30 14:38:18.024932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.024874 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4bft"] Oct 30 14:38:18.318538 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.318459 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd63488-9f69-4048-ae51-e8c9d1df74cb" path="/var/lib/kubelet/pods/ddd63488-9f69-4048-ae51-e8c9d1df74cb/volumes" Oct 30 14:38:18.913659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.913625 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs"] Oct 30 14:38:18.914582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.914560 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ddd63488-9f69-4048-ae51-e8c9d1df74cb" containerName="discovery" Oct 30 14:38:18.914582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.914582 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd63488-9f69-4048-ae51-e8c9d1df74cb" containerName="discovery" Oct 30 14:38:18.914802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.914788 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd63488-9f69-4048-ae51-e8c9d1df74cb" containerName="discovery" Oct 30 14:38:18.920371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.920337 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:18.939555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.939532 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs"] Oct 30 14:38:18.949510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:18.949486 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-9e3aa0-openshift-default-dockercfg-n4l2x\"" Oct 30 14:38:19.027129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027094 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtbcm\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-kube-api-access-vtbcm\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.027314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027134 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-data\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.027314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027212 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-envoy\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.027398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027378 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9054b512-4a30-4209-baf0-3f0ccd66a73d-istiod-ca-cert\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.027435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027426 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-certs\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.027469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027445 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-token\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.027557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027542 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.027595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027575 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-credential-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.027633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.027608 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-podinfo\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129344 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-certs\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129384 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-token\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129413 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129438 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-credential-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129473 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-podinfo\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129568 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtbcm\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-kube-api-access-vtbcm\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129596 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-data\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129645 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-envoy\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129754 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9054b512-4a30-4209-baf0-3f0ccd66a73d-istiod-ca-cert\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.129994 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129849 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-certs\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.130051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.129992 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.130111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.130091 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-data\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.130171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.130147 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-credential-socket\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.130606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.130580 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9054b512-4a30-4209-baf0-3f0ccd66a73d-istiod-ca-cert\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.132227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.132174 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-envoy\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.132392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.132375 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-podinfo\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.156843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.156811 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-token\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.156958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.156942 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtbcm\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-kube-api-access-vtbcm\") pod \"e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.238708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.238676 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:19.483594 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.483556 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5"] Oct 30 14:38:19.488475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.488443 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:19.509142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.506723 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0e8b9\"" Oct 30 14:38:19.509142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.507559 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0e8b9-dockercfg-ztrwp\"" Oct 30 14:38:19.511752 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.511727 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5"] Oct 30 14:38:19.598896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.598264 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf"] Oct 30 14:38:19.611580 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:38:19.605656 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9054b512_4a30_4209_baf0_3f0ccd66a73d.slice/crio-fbbe5503504b82b79ce9f234dc12ba9e9e8d5981022ac80fb7903c8e7122b7cb WatchSource:0}: Error finding container fbbe5503504b82b79ce9f234dc12ba9e9e8d5981022ac80fb7903c8e7122b7cb: Status 404 returned error can't find the container with id fbbe5503504b82b79ce9f234dc12ba9e9e8d5981022ac80fb7903c8e7122b7cb Oct 30 14:38:19.678723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.678679 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc42z\" (UniqueName: \"kubernetes.io/projected/5a692131-bb69-4e30-9069-8aa7c017d176-kube-api-access-zc42z\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5\" (UID: \"5a692131-bb69-4e30-9069-8aa7c017d176\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:19.679145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.679119 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a692131-bb69-4e30-9069-8aa7c017d176-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5\" (UID: \"5a692131-bb69-4e30-9069-8aa7c017d176\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:19.700783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.700720 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:38:19.701145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.701113 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:38:19.701470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.701439 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:38:19.709322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.709297 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:19.709538 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.709516 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs"] Oct 30 14:38:19.709613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.709600 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf"] Oct 30 14:38:19.781287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.781252 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc42z\" (UniqueName: \"kubernetes.io/projected/5a692131-bb69-4e30-9069-8aa7c017d176-kube-api-access-zc42z\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5\" (UID: \"5a692131-bb69-4e30-9069-8aa7c017d176\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:19.781407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.781380 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a692131-bb69-4e30-9069-8aa7c017d176-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5\" (UID: \"5a692131-bb69-4e30-9069-8aa7c017d176\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:19.782220 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.782174 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a692131-bb69-4e30-9069-8aa7c017d176-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5\" (UID: \"5a692131-bb69-4e30-9069-8aa7c017d176\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:19.808678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.808648 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc42z\" (UniqueName: \"kubernetes.io/projected/5a692131-bb69-4e30-9069-8aa7c017d176-kube-api-access-zc42z\") pod \"payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5\" (UID: \"5a692131-bb69-4e30-9069-8aa7c017d176\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:19.811472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.811445 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:19.883414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.883301 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf\" (UID: \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:19.883512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.883423 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2vm\" (UniqueName: \"kubernetes.io/projected/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-kube-api-access-2d2vm\") pod \"payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf\" (UID: \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:19.979780 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.979738 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" event={"ID":"9054b512-4a30-4209-baf0-3f0ccd66a73d","Type":"ContainerStarted","Data":"48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595"} Oct 30 14:38:19.980310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.979812 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" event={"ID":"9054b512-4a30-4209-baf0-3f0ccd66a73d","Type":"ContainerStarted","Data":"fbbe5503504b82b79ce9f234dc12ba9e9e8d5981022ac80fb7903c8e7122b7cb"} Oct 30 14:38:19.984699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.984668 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf\" (UID: \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:19.984829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.984714 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2vm\" (UniqueName: \"kubernetes.io/projected/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-kube-api-access-2d2vm\") pod \"payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf\" (UID: \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:19.985467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:19.985446 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf\" (UID: \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:20.011359 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.011333 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2vm\" (UniqueName: \"kubernetes.io/projected/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-kube-api-access-2d2vm\") pod \"payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf\" (UID: \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:20.027740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.027663 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:20.033077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.033039 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podStartSLOduration=2.033027218 podStartE2EDuration="2.033027218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:20.030674026 +0000 UTC m=+2929.831179470" watchObservedRunningTime="2026-10-30 14:38:20.033027218 +0000 UTC m=+2929.833532661" Oct 30 14:38:20.126945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.126465 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5"] Oct 30 14:38:20.130839 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:38:20.130665 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a692131_bb69_4e30_9069_8aa7c017d176.slice/crio-71b04e20b3860cbb53e852e33a6f672aa7c39de9798db57ca4e698d399697f80 WatchSource:0}: Error finding container 71b04e20b3860cbb53e852e33a6f672aa7c39de9798db57ca4e698d399697f80: Status 404 returned error can't find the container with id 71b04e20b3860cbb53e852e33a6f672aa7c39de9798db57ca4e698d399697f80 Oct 30 14:38:20.239634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.239605 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:20.241032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.241004 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 14:38:20.241157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.241130 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 14:38:20.336896 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:38:20.336863 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c09e8f_cac7_465b_9cc6_ae4ed40212d8.slice/crio-d9736590d01b13858f71123f35372132743e1e7b967ae33e183e5c6ace1901c8 WatchSource:0}: Error finding container d9736590d01b13858f71123f35372132743e1e7b967ae33e183e5c6ace1901c8: Status 404 returned error can't find the container with id d9736590d01b13858f71123f35372132743e1e7b967ae33e183e5c6ace1901c8 Oct 30 14:38:20.338301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.338277 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf"] Oct 30 14:38:20.989125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.989087 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" event={"ID":"38c09e8f-cac7-465b-9cc6-ae4ed40212d8","Type":"ContainerStarted","Data":"d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a"} Oct 30 14:38:20.989632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.989159 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:20.989632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.989200 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" event={"ID":"38c09e8f-cac7-465b-9cc6-ae4ed40212d8","Type":"ContainerStarted","Data":"d9736590d01b13858f71123f35372132743e1e7b967ae33e183e5c6ace1901c8"} Oct 30 14:38:20.990771 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.990741 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" event={"ID":"5a692131-bb69-4e30-9069-8aa7c017d176","Type":"ContainerStarted","Data":"691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97"} Oct 30 14:38:20.990911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.990789 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" event={"ID":"5a692131-bb69-4e30-9069-8aa7c017d176","Type":"ContainerStarted","Data":"71b04e20b3860cbb53e852e33a6f672aa7c39de9798db57ca4e698d399697f80"} Oct 30 14:38:20.990911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:20.990874 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:21.038931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:21.038879 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" podStartSLOduration=2.038865529 podStartE2EDuration="2.038865529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:21.035541718 +0000 UTC m=+2930.836047161" watchObservedRunningTime="2026-10-30 14:38:21.038865529 +0000 UTC m=+2930.839371004" Oct 30 14:38:21.084863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:21.084815 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" podStartSLOduration=2.084802793 podStartE2EDuration="2.084802793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:21.082528409 +0000 UTC m=+2930.883033854" watchObservedRunningTime="2026-10-30 14:38:21.084802793 +0000 UTC m=+2930.885308270" Oct 30 14:38:21.239984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:21.239900 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 14:38:21.239984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:21.239959 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 14:38:22.239339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:22.239304 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 14:38:22.239730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:22.239371 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 14:38:23.239171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:23.239130 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 14:38:23.239371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:23.239210 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 14:38:24.239263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:24.239225 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 14:38:24.239464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:24.239283 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 14:38:25.239839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:25.239805 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 14:38:25.240290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:25.239866 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 14:38:26.239785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:26.239746 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 14:38:26.239976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:26.239810 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 14:38:27.409302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:27.409267 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:27.410253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:27.410226 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:27.411297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:27.411270 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:38:32.006118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:32.006090 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:38:32.006578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:32.006504 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:38:33.644388 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:38:33.644354 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:38:38.895910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:38.895872 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72"] Oct 30 14:38:38.896492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:38.896093 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" containerID="cri-o://c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86" gracePeriod=30 Oct 30 14:38:39.792836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.792803 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm"] Oct 30 14:38:39.798933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.798913 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.808950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.808924 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-token\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.809043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.808958 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-certs\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.809043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.808986 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.809140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.809092 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-data\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.809182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.809146 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-podinfo\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.809182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.809176 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jszf8\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-kube-api-access-jszf8\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.809272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.809216 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istiod-ca-cert\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.809437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.809408 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-envoy\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.809493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.809482 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-credential-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.819791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.819755 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm"] Oct 30 14:38:39.828124 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.828095 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f1f206-openshift-default-dockercfg-kf8p5\"" Oct 30 14:38:39.910974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.910937 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-data\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911011 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-podinfo\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911064 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jszf8\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-kube-api-access-jszf8\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911118 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istiod-ca-cert\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911237 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-envoy\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911288 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-credential-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911413 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-token\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911440 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-certs\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911455 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-data\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911481 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.911935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.911844 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.912128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.912107 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-certs\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.912240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.912142 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istiod-ca-cert\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.912341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.912317 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-credential-socket\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.915401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.915252 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-envoy\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.916481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.916455 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-podinfo\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.944517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.944486 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-token\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:39.944660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:39.944638 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jszf8\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-kube-api-access-jszf8\") pod \"e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:40.091060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.090987 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg"] Oct 30 14:38:40.096877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.096859 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:40.113418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.113391 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:40.114076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.114017 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbrz\" (UniqueName: \"kubernetes.io/projected/cdf724df-3ede-412e-814d-5feb227da664-kube-api-access-2sbrz\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg\" (UID: \"cdf724df-3ede-412e-814d-5feb227da664\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:40.114368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.114344 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf724df-3ede-412e-814d-5feb227da664-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg\" (UID: \"cdf724df-3ede-412e-814d-5feb227da664\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:40.117876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.117838 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-460b0b-dockercfg-k76k4\"" Oct 30 14:38:40.118332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.118311 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg"] Oct 30 14:38:40.120721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.120701 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-460b0b\"" Oct 30 14:38:40.165635 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.165604 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg"] Oct 30 14:38:40.170496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.170459 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:40.225215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.221610 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf724df-3ede-412e-814d-5feb227da664-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg\" (UID: \"cdf724df-3ede-412e-814d-5feb227da664\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:40.225215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.221705 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbrz\" (UniqueName: \"kubernetes.io/projected/cdf724df-3ede-412e-814d-5feb227da664-kube-api-access-2sbrz\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg\" (UID: \"cdf724df-3ede-412e-814d-5feb227da664\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:40.225215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.221757 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4515b-90dd-4418-83e6-879fed176399-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg\" (UID: \"fad4515b-90dd-4418-83e6-879fed176399\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:40.225215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.221805 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbmd7\" (UniqueName: \"kubernetes.io/projected/fad4515b-90dd-4418-83e6-879fed176399-kube-api-access-hbmd7\") pod \"payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg\" (UID: \"fad4515b-90dd-4418-83e6-879fed176399\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:40.225215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.222760 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf724df-3ede-412e-814d-5feb227da664-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg\" (UID: \"cdf724df-3ede-412e-814d-5feb227da664\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:40.232153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.229729 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg"] Oct 30 14:38:40.290083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.288932 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbrz\" (UniqueName: \"kubernetes.io/projected/cdf724df-3ede-412e-814d-5feb227da664-kube-api-access-2sbrz\") pod \"payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg\" (UID: \"cdf724df-3ede-412e-814d-5feb227da664\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:40.323162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.323135 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbmd7\" (UniqueName: \"kubernetes.io/projected/fad4515b-90dd-4418-83e6-879fed176399-kube-api-access-hbmd7\") pod \"payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg\" (UID: \"fad4515b-90dd-4418-83e6-879fed176399\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:40.323526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.323497 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4515b-90dd-4418-83e6-879fed176399-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg\" (UID: \"fad4515b-90dd-4418-83e6-879fed176399\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:40.324224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.324171 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4515b-90dd-4418-83e6-879fed176399-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg\" (UID: \"fad4515b-90dd-4418-83e6-879fed176399\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:40.350730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.350665 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbmd7\" (UniqueName: \"kubernetes.io/projected/fad4515b-90dd-4418-83e6-879fed176399-kube-api-access-hbmd7\") pod \"payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg\" (UID: \"fad4515b-90dd-4418-83e6-879fed176399\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:40.412886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.412859 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:40.491213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.483815 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm"] Oct 30 14:38:40.491213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.488159 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:38:40.491213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.488263 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:38:40.491213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.488302 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:38:40.605791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.605707 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:40.739444 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:38:40.739399 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf724df_3ede_412e_814d_5feb227da664.slice/crio-51e08e82e9d79ab4c3ed257d33e9077f727c8676a7a85d15d9296749e3078849 WatchSource:0}: Error finding container 51e08e82e9d79ab4c3ed257d33e9077f727c8676a7a85d15d9296749e3078849: Status 404 returned error can't find the container with id 51e08e82e9d79ab4c3ed257d33e9077f727c8676a7a85d15d9296749e3078849 Oct 30 14:38:40.741604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.741571 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg"] Oct 30 14:38:40.939611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:40.939584 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg"] Oct 30 14:38:41.275608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.275507 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" event={"ID":"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7","Type":"ContainerStarted","Data":"58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee"} Oct 30 14:38:41.275608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.275567 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" event={"ID":"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7","Type":"ContainerStarted","Data":"28b2112a0a6430172beb228c3e340fed56db1318e40a937b5fc990ed389fa1c9"} Oct 30 14:38:41.277323 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.277293 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" event={"ID":"cdf724df-3ede-412e-814d-5feb227da664","Type":"ContainerStarted","Data":"b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f"} Oct 30 14:38:41.277485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.277345 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" event={"ID":"cdf724df-3ede-412e-814d-5feb227da664","Type":"ContainerStarted","Data":"51e08e82e9d79ab4c3ed257d33e9077f727c8676a7a85d15d9296749e3078849"} Oct 30 14:38:41.277485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.277391 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:41.279111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.279081 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" event={"ID":"fad4515b-90dd-4418-83e6-879fed176399","Type":"ContainerStarted","Data":"f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2"} Oct 30 14:38:41.279234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.279133 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:41.279234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.279152 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" event={"ID":"fad4515b-90dd-4418-83e6-879fed176399","Type":"ContainerStarted","Data":"9cc1375b7b299290a7b1776825fc1b333d6d9ec75755f222d95e4962dd16145e"} Oct 30 14:38:41.326521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.326464 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" podStartSLOduration=2.326449352 podStartE2EDuration="2.326449352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:41.324428595 +0000 UTC m=+2951.124934039" watchObservedRunningTime="2026-10-30 14:38:41.326449352 +0000 UTC m=+2951.126954795" Oct 30 14:38:41.370934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.370880 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" podStartSLOduration=1.370866952 podStartE2EDuration="1.370866952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:41.37042232 +0000 UTC m=+2951.170927764" watchObservedRunningTime="2026-10-30 14:38:41.370866952 +0000 UTC m=+2951.171372405" Oct 30 14:38:41.429014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:41.428960 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" podStartSLOduration=1.428946684 podStartE2EDuration="1.428946684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:41.425399899 +0000 UTC m=+2951.225905343" watchObservedRunningTime="2026-10-30 14:38:41.428946684 +0000 UTC m=+2951.229452127" Oct 30 14:38:42.113827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:42.113787 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:42.115356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:42.115334 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 14:38:42.115419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:42.115384 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 14:38:43.114253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:43.114211 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 14:38:43.114765 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:43.114278 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 14:38:44.128221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.124583 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 14:38:44.128221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.124655 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 14:38:44.238405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.238371 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:38:44.293785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.293754 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-socket\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.293967 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.293802 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-certs\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.293967 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.293835 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-envoy\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.293967 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.293921 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-credential-socket\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.294128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294008 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-token\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-token\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.294128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294031 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-socket" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:44.294128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294103 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istiod-ca-cert\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.294435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294167 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-podinfo\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.294435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294182 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-certs" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:44.294435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294270 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-kube-api-access-9hhqs\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-kube-api-access-9hhqs\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.294435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294302 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-credential-socket" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:44.294435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294324 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-data\") pod \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\" (UID: \"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78\") " Oct 30 14:38:44.294723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294503 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istiod-ca-cert" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:38:44.294723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294616 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-data" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:44.294928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294905 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.294928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294930 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.295068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294940 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.295068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294948 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.295068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.294957 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.296657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.296631 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-podinfo" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:38:44.296818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.296796 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-envoy" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:38:44.296891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.296841 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-token" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:44.296939 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.296900 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-kube-api-access-9hhqs" pod "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" (UID: "e4aef07d-fc7a-4a47-8a50-4a35d9f19a78"). InnerVolumeSpecName "kube-api-access-9hhqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:44.304595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.304568 5824 generic.go:356] "Generic (PLEG): container finished" podID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerID="c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86" exitCode=0 Oct 30 14:38:44.304695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.304617 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" Oct 30 14:38:44.304695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.304648 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" event={"ID":"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78","Type":"ContainerDied","Data":"c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86"} Oct 30 14:38:44.304792 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.304706 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72" event={"ID":"e4aef07d-fc7a-4a47-8a50-4a35d9f19a78","Type":"ContainerDied","Data":"50b1976367b045aae7a1e17d9e6c12445a6b670dcba047dfb034e2f1eccafe47"} Oct 30 14:38:44.304792 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.304738 5824 scope.go:122] "RemoveContainer" containerID="c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86" Oct 30 14:38:44.379818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.379791 5824 scope.go:122] "RemoveContainer" containerID="c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86" Oct 30 14:38:44.380870 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:38:44.380849 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86\": container with ID starting with c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86 not found: ID does not exist" containerID="c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86" Oct 30 14:38:44.380998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.380980 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86"} err="failed to get container status \"c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86\": rpc error: code = NotFound desc = could not find container \"c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86\": container with ID starting with c5e1ef08945de7031d5d8fe6236f481ac487d9be3a687fdd4d6e94d1a9bfdf86 not found: ID does not exist" Oct 30 14:38:44.397993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.397546 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hhqs\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-kube-api-access-9hhqs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.397993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.397578 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.397993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.397594 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.397993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.397607 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:44.430921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.430885 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72"] Oct 30 14:38:44.435362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.435334 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6cf72"] Oct 30 14:38:44.591956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.591869 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9591-e484aa"] Oct 30 14:38:44.592751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.592727 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" Oct 30 14:38:44.592891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.592750 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" Oct 30 14:38:44.592991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.592975 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" containerName="istio-proxy" Oct 30 14:38:44.598324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.598303 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-e484aa" Oct 30 14:38:44.615447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.615424 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-e484aa"] Oct 30 14:38:44.617258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.617228 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q95l7\"" Oct 30 14:38:44.701708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.701675 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw8pr\" (UniqueName: \"kubernetes.io/projected/f5f570ba-6081-4f2a-b149-fe06890fe29e-kube-api-access-vw8pr\") pod \"test-curl-9591-e484aa\" (UID: \"f5f570ba-6081-4f2a-b149-fe06890fe29e\") " pod="opendatahub/test-curl-9591-e484aa" Oct 30 14:38:44.803111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.803070 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw8pr\" (UniqueName: \"kubernetes.io/projected/f5f570ba-6081-4f2a-b149-fe06890fe29e-kube-api-access-vw8pr\") pod \"test-curl-9591-e484aa\" (UID: \"f5f570ba-6081-4f2a-b149-fe06890fe29e\") " pod="opendatahub/test-curl-9591-e484aa" Oct 30 14:38:44.850528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.841616 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw8pr\" (UniqueName: \"kubernetes.io/projected/f5f570ba-6081-4f2a-b149-fe06890fe29e-kube-api-access-vw8pr\") pod \"test-curl-9591-e484aa\" (UID: \"f5f570ba-6081-4f2a-b149-fe06890fe29e\") " pod="opendatahub/test-curl-9591-e484aa" Oct 30 14:38:44.911347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:44.911309 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-e484aa" Oct 30 14:38:45.114643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:45.114555 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 14:38:45.114643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:45.114614 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 14:38:45.269061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:45.269026 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9591-e484aa"] Oct 30 14:38:45.271657 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:38:45.271620 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f570ba_6081_4f2a_b149_fe06890fe29e.slice/crio-d3d7cd92a630d9c0dba84f727fb9beb9070cdf90d8384dfc7a59d2f38543ed6d WatchSource:0}: Error finding container d3d7cd92a630d9c0dba84f727fb9beb9070cdf90d8384dfc7a59d2f38543ed6d: Status 404 returned error can't find the container with id d3d7cd92a630d9c0dba84f727fb9beb9070cdf90d8384dfc7a59d2f38543ed6d Oct 30 14:38:45.273740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:45.273602 5824 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:38:45.327407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:45.326923 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-e484aa" event={"ID":"f5f570ba-6081-4f2a-b149-fe06890fe29e","Type":"ContainerStarted","Data":"d3d7cd92a630d9c0dba84f727fb9beb9070cdf90d8384dfc7a59d2f38543ed6d"} Oct 30 14:38:46.114310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:46.114275 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 14:38:46.114521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:46.114338 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 14:38:46.319494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:46.319454 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4aef07d-fc7a-4a47-8a50-4a35d9f19a78" path="/var/lib/kubelet/pods/e4aef07d-fc7a-4a47-8a50-4a35d9f19a78/volumes" Oct 30 14:38:47.113890 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:47.113830 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 14:38:47.114093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:47.113928 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 14:38:48.119893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:48.119865 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:48.120338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:48.120277 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:48.120981 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:48.120960 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:38:48.350454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:48.350365 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9591-e484aa" event={"ID":"f5f570ba-6081-4f2a-b149-fe06890fe29e","Type":"ContainerStarted","Data":"77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a"} Oct 30 14:38:48.393561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:48.393490 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9591-e484aa" podStartSLOduration=1.578953727 podStartE2EDuration="4.393474868s" totalImagesPullingTime="2.814521141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:38:48.392672401 +0000 UTC m=+2958.193177844" watchObservedRunningTime="2026-10-30 14:38:48.393474868 +0000 UTC m=+2958.193980312" Oct 30 14:38:49.348040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:49.348005 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9591-e484aa"] Oct 30 14:38:49.354415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:49.354384 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9591-e484aa"] Oct 30 14:38:50.370110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:50.370053 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9591-e484aa" podUID="f5f570ba-6081-4f2a-b149-fe06890fe29e" containerName="test-curl-9591-e484aa" containerID="cri-o://77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a" gracePeriod=2 Oct 30 14:38:52.291294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:52.291264 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:38:52.291724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:52.291331 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:38:52.607798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:52.607769 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-e484aa" Oct 30 14:38:52.744810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:52.744771 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f570ba-6081-4f2a-b149-fe06890fe29e-kube-api-access-vw8pr\" (UniqueName: \"kubernetes.io/projected/f5f570ba-6081-4f2a-b149-fe06890fe29e-kube-api-access-vw8pr\") pod \"f5f570ba-6081-4f2a-b149-fe06890fe29e\" (UID: \"f5f570ba-6081-4f2a-b149-fe06890fe29e\") " Oct 30 14:38:52.746862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:52.746840 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f570ba-6081-4f2a-b149-fe06890fe29e-kube-api-access-vw8pr" pod "f5f570ba-6081-4f2a-b149-fe06890fe29e" (UID: "f5f570ba-6081-4f2a-b149-fe06890fe29e"). InnerVolumeSpecName "kube-api-access-vw8pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:38:52.846406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:52.846319 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw8pr\" (UniqueName: \"kubernetes.io/projected/f5f570ba-6081-4f2a-b149-fe06890fe29e-kube-api-access-vw8pr\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:38:53.400480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:53.400442 5824 generic.go:356] "Generic (PLEG): container finished" podID="f5f570ba-6081-4f2a-b149-fe06890fe29e" containerID="77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a" exitCode=137 Oct 30 14:38:53.400919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:53.400497 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9591-e484aa" Oct 30 14:38:53.400919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:53.400559 5824 scope.go:122] "RemoveContainer" containerID="77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a" Oct 30 14:38:53.411644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:53.411623 5824 scope.go:122] "RemoveContainer" containerID="77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a" Oct 30 14:38:53.411944 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:38:53.411923 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a\": container with ID starting with 77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a not found: ID does not exist" containerID="77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a" Oct 30 14:38:53.411994 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:53.411953 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a"} err="failed to get container status \"77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a\": rpc error: code = NotFound desc = could not find container \"77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a\": container with ID starting with 77cfef962df37ef3c732a7ad2c86b708e67b39baa4536819d3050b306865cd4a not found: ID does not exist" Oct 30 14:38:53.424900 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:38:53.424860 5824 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9591-e484aa\" is forbidden: User \"system:node:ip-10-0-131-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-200.ec2.internal' and this object" podUID="f5f570ba-6081-4f2a-b149-fe06890fe29e" pod="opendatahub/test-curl-9591-e484aa" Oct 30 14:38:54.320518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:38:54.320487 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f570ba-6081-4f2a-b149-fe06890fe29e" path="/var/lib/kubelet/pods/f5f570ba-6081-4f2a-b149-fe06890fe29e/volumes" Oct 30 14:39:01.304093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.304057 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m"] Oct 30 14:39:01.305151 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.305129 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f5f570ba-6081-4f2a-b149-fe06890fe29e" containerName="test-curl-9591-e484aa" Oct 30 14:39:01.305235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.305155 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f570ba-6081-4f2a-b149-fe06890fe29e" containerName="test-curl-9591-e484aa" Oct 30 14:39:01.305447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.305429 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f570ba-6081-4f2a-b149-fe06890fe29e" containerName="test-curl-9591-e484aa" Oct 30 14:39:01.309121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.309102 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.335236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.333304 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m"] Oct 30 14:39:01.342671 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.342635 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-33c209-openshift-default-dockercfg-dvj8c\"" Oct 30 14:39:01.374781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.374722 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-podinfo\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.374961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.374856 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npk5\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-kube-api-access-8npk5\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.374961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.374925 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-envoy\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.375316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.375034 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f88aaf-4caa-47d3-badd-9a26b264052b-istiod-ca-cert\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.375316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.375135 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-data\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.375316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.375221 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-credential-socket\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.375316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.375253 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-token\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.375316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.375278 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-socket\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.375316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.375295 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-certs\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.476469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.476432 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-socket\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.476660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.476490 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-certs\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.476660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.476651 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-podinfo\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.476794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.476719 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npk5\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-kube-api-access-8npk5\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.476794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.476780 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-envoy\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.476932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.476916 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f88aaf-4caa-47d3-badd-9a26b264052b-istiod-ca-cert\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.476984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.476947 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-data\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.477043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.477016 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-credential-socket\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.477098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.477059 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-token\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.477283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.477248 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-socket\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.477706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.477663 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-certs\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.477802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.477711 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-data\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.477802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.477735 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-credential-socket\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.477802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.477746 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f88aaf-4caa-47d3-badd-9a26b264052b-istiod-ca-cert\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.479693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.479661 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-podinfo\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.480099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.480071 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-envoy\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.503235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.503167 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-token\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.503403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.503374 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npk5\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-kube-api-access-8npk5\") pod \"e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.623650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.623576 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:01.956280 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:39:01.956245 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f88aaf_4caa_47d3_badd_9a26b264052b.slice/crio-5d9556474ee09bcee6fe6fe066f0ec018cbcd70515283e128514548cec37b918 WatchSource:0}: Error finding container 5d9556474ee09bcee6fe6fe066f0ec018cbcd70515283e128514548cec37b918: Status 404 returned error can't find the container with id 5d9556474ee09bcee6fe6fe066f0ec018cbcd70515283e128514548cec37b918 Oct 30 14:39:01.956652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.956628 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m"] Oct 30 14:39:01.959255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.959225 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:01.959335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.959285 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:01.959335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:01.959318 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:02.203357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.203282 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc"] Oct 30 14:39:02.207968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.207935 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.248960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.248924 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc"] Oct 30 14:39:02.256468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.256437 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-396d70-openshift-default-dockercfg-pnzlz\"" Oct 30 14:39:02.289634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289068 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-credential-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.289634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289169 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-certs\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.289634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289272 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6zb\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-kube-api-access-sw6zb\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.289634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289391 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-token\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.289634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289488 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-envoy\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.289634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289548 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-data\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.289634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289623 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-podinfo\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.289996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289661 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istiod-ca-cert\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.289996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.289686 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.391535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.391495 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-podinfo\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.391978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.391553 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istiod-ca-cert\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.391978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.391665 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.391978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.391794 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-credential-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.391978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.391834 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-certs\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.391978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.391860 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6zb\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-kube-api-access-sw6zb\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.391978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.391943 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-token\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.392452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.392027 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-envoy\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.392452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.392065 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-data\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.392452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.392102 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.392452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.392228 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-certs\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.392452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.392243 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-credential-socket\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.392452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.392383 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istiod-ca-cert\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.392679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.392505 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-data\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.394215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.394160 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-podinfo\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.394318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.394165 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-envoy\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.418608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.418579 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-token\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.418715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.418649 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6zb\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-kube-api-access-sw6zb\") pod \"e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.469362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.469262 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" event={"ID":"06f88aaf-4caa-47d3-badd-9a26b264052b","Type":"ContainerStarted","Data":"a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c"} Oct 30 14:39:02.469362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.469324 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" event={"ID":"06f88aaf-4caa-47d3-badd-9a26b264052b","Type":"ContainerStarted","Data":"5d9556474ee09bcee6fe6fe066f0ec018cbcd70515283e128514548cec37b918"} Oct 30 14:39:02.523525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.523483 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:02.525481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.525427 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" podStartSLOduration=1.5254099779999999 podStartE2EDuration="1.525409978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:02.523250458 +0000 UTC m=+2972.323755919" watchObservedRunningTime="2026-10-30 14:39:02.525409978 +0000 UTC m=+2972.325915423" Oct 30 14:39:02.624390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.624354 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:02.866753 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:39:02.866722 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7cbdf7_c946_4b38_8716_4f52cc8bc9b0.slice/crio-e91341e55f3b4874d9b0b874b2beb878e6c7fd968f41d39349eeebec29c1bf48 WatchSource:0}: Error finding container e91341e55f3b4874d9b0b874b2beb878e6c7fd968f41d39349eeebec29c1bf48: Status 404 returned error can't find the container with id e91341e55f3b4874d9b0b874b2beb878e6c7fd968f41d39349eeebec29c1bf48 Oct 30 14:39:02.870000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.869976 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc"] Oct 30 14:39:02.870262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.870223 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:02.870322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.870302 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:02.870364 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:02.870335 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:03.482242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:03.482183 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" event={"ID":"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0","Type":"ContainerStarted","Data":"73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd"} Oct 30 14:39:03.482741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:03.482261 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" event={"ID":"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0","Type":"ContainerStarted","Data":"e91341e55f3b4874d9b0b874b2beb878e6c7fd968f41d39349eeebec29c1bf48"} Oct 30 14:39:03.524241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:03.524212 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:03.526019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:03.525993 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 14:39:03.526120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:03.526043 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 14:39:03.572964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:03.572906 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" podStartSLOduration=1.572890631 podStartE2EDuration="1.572890631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:03.558727653 +0000 UTC m=+2973.359233097" watchObservedRunningTime="2026-10-30 14:39:03.572890631 +0000 UTC m=+2973.373396072" Oct 30 14:39:03.632595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:03.632566 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:03.669235 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:39:03.667450 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:39:04.490919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:04.490882 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:04.491602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:04.491583 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:39:04.524675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:04.524644 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 14:39:04.524983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:04.524694 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 14:39:05.524270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:05.524233 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 14:39:05.524837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:05.524330 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 14:39:06.030205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.030158 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m"] Oct 30 14:39:06.035016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.034994 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.078217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.076792 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m"] Oct 30 14:39:06.126821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.126783 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-75443e-openshift-default-dockercfg-r7cs9\"" Oct 30 14:39:06.176930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.176884 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqxd\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-kube-api-access-5dqxd\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.177100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.177047 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.177152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.177106 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.177281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.177253 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.177353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.177333 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cd96b37-6ca8-4034-8c64-b99ab1652451-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.177492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.177463 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.177603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.177580 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.177694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.177675 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.177746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.177734 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.279606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.279572 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.279793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.279664 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqxd\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-kube-api-access-5dqxd\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.279793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.279732 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.279793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.279783 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.279993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.279824 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.279993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.279869 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cd96b37-6ca8-4034-8c64-b99ab1652451-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.279993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.279984 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.280164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.280029 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-certs\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.280164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.280051 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.280164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.280051 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-data\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.280164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.280144 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.280575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.280524 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.280575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.280535 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-credential-socket\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.280752 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.280734 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cd96b37-6ca8-4034-8c64-b99ab1652451-istiod-ca-cert\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.282348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.282315 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-envoy\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.282842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.282821 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-podinfo\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.312626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.312595 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-token\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.312783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.312657 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqxd\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-kube-api-access-5dqxd\") pod \"e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.377039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.377000 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:06.526099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.526057 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 14:39:06.526498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.526131 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 14:39:06.738101 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:39:06.738071 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd96b37_6ca8_4034_8c64_b99ab1652451.slice/crio-eb9b36dba9c7e76179b0565a9a02e1f3ea2186141dc48af79842177c270d8521 WatchSource:0}: Error finding container eb9b36dba9c7e76179b0565a9a02e1f3ea2186141dc48af79842177c270d8521: Status 404 returned error can't find the container with id eb9b36dba9c7e76179b0565a9a02e1f3ea2186141dc48af79842177c270d8521 Oct 30 14:39:06.739086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.739060 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m"] Oct 30 14:39:06.741173 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.741139 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:06.741283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.741242 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:06.765267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:06.741286 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:39:07.524171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:07.524135 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 14:39:07.524357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:07.524225 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 14:39:07.525495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:07.525464 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" event={"ID":"8cd96b37-6ca8-4034-8c64-b99ab1652451","Type":"ContainerStarted","Data":"5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11"} Oct 30 14:39:07.525622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:07.525529 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" event={"ID":"8cd96b37-6ca8-4034-8c64-b99ab1652451","Type":"ContainerStarted","Data":"eb9b36dba9c7e76179b0565a9a02e1f3ea2186141dc48af79842177c270d8521"} Oct 30 14:39:07.616044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:07.615981 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podStartSLOduration=2.615960557 podStartE2EDuration="2.615960557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:39:07.615123515 +0000 UTC m=+2977.415628952" watchObservedRunningTime="2026-10-30 14:39:07.615960557 +0000 UTC m=+2977.416466003" Oct 30 14:39:08.383223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:08.382770 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:08.383223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:08.382939 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 14:39:08.383223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:08.382992 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 14:39:08.524628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:08.524594 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Oct 30 14:39:08.524891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:08.524862 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Oct 30 14:39:09.377906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:09.377853 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 14:39:09.378453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:09.377936 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 14:39:09.812930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:09.812894 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:09.814080 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:09.814055 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:09.815167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:09.815142 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:39:10.377593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:10.377558 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 14:39:10.377873 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:10.377615 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 14:39:11.377483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:11.377390 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 14:39:11.377960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:11.377475 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 14:39:12.377361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:12.377330 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 14:39:12.377572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:12.377402 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 14:39:13.378019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:13.377978 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 14:39:13.378459 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:13.378053 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 14:39:14.383260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:14.383225 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:14.383784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:14.383758 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:14.384747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:14.384719 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:39:33.644494 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:39:33.644454 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:39:45.036740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:45.036706 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78"] Oct 30 14:39:45.050071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:45.050042 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4059a2d7-ghz78"] Oct 30 14:39:46.321238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:46.321184 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a109044-de80-4edf-b7e8-cc82422eb9d8" path="/var/lib/kubelet/pods/8a109044-de80-4edf-b7e8-cc82422eb9d8/volumes" Oct 30 14:39:55.051478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:55.051431 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2"] Oct 30 14:39:55.064908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:55.064857 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed2673e5-gvpq2"] Oct 30 14:39:56.320675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:39:56.320643 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2022b464-958d-4738-8a24-5661fa6646f8" path="/var/lib/kubelet/pods/2022b464-958d-4738-8a24-5661fa6646f8/volumes" Oct 30 14:40:03.643935 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:40:03.643884 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:40:04.736524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.736485 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679"] Oct 30 14:40:04.742255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.742233 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:04.757923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.757892 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679"] Oct 30 14:40:04.803054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.803025 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a1854b-dockercfg-dt6nt\"" Oct 30 14:40:04.807648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.807629 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a1854b\"" Oct 30 14:40:04.823803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.823763 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knd46\" (UniqueName: \"kubernetes.io/projected/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-kube-api-access-knd46\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679\" (UID: \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:04.824423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.824390 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679\" (UID: \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:04.930363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.925675 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679\" (UID: \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:04.930749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.930719 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knd46\" (UniqueName: \"kubernetes.io/projected/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-kube-api-access-knd46\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679\" (UID: \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:04.934240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.932202 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679\" (UID: \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:04.958221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:04.958178 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knd46\" (UniqueName: \"kubernetes.io/projected/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-kube-api-access-knd46\") pod \"payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679\" (UID: \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:05.057939 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:05.057855 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:05.177494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:05.176595 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf"] Oct 30 14:40:05.208589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:05.208555 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5"] Oct 30 14:40:05.400405 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:40:05.400371 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88d8cdb_c0bf_4390_894e_ce91cfd1414b.slice/crio-31dac9a2c830bd28b612da3e788ba316e60f47269cf20f1a9619988b7fbb6bb5 WatchSource:0}: Error finding container 31dac9a2c830bd28b612da3e788ba316e60f47269cf20f1a9619988b7fbb6bb5: Status 404 returned error can't find the container with id 31dac9a2c830bd28b612da3e788ba316e60f47269cf20f1a9619988b7fbb6bb5 Oct 30 14:40:05.401467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:05.401445 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679"] Oct 30 14:40:06.054478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:06.054441 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" event={"ID":"e88d8cdb-c0bf-4390-894e-ce91cfd1414b","Type":"ContainerStarted","Data":"bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5"} Oct 30 14:40:06.054912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:06.054497 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" event={"ID":"e88d8cdb-c0bf-4390-894e-ce91cfd1414b","Type":"ContainerStarted","Data":"31dac9a2c830bd28b612da3e788ba316e60f47269cf20f1a9619988b7fbb6bb5"} Oct 30 14:40:06.054912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:06.054703 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:06.108950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:06.108887 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" podStartSLOduration=2.108867062 podStartE2EDuration="2.108867062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:06.105825432 +0000 UTC m=+3035.906330877" watchObservedRunningTime="2026-10-30 14:40:06.108867062 +0000 UTC m=+3035.909372509" Oct 30 14:40:09.901267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:09.901228 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv"] Oct 30 14:40:09.906251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:09.906226 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:09.945573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:09.944387 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv"] Oct 30 14:40:09.952564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:09.952525 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6176c3-dockercfg-d9c25\"" Oct 30 14:40:09.986229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:09.983962 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6176c3\"" Oct 30 14:40:10.068208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.062912 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c4642c-5571-4239-aab5-0c5a254102a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv\" (UID: \"87c4642c-5571-4239-aab5-0c5a254102a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:10.068208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.063064 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g49\" (UniqueName: \"kubernetes.io/projected/87c4642c-5571-4239-aab5-0c5a254102a3-kube-api-access-74g49\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv\" (UID: \"87c4642c-5571-4239-aab5-0c5a254102a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:10.164281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.164182 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c4642c-5571-4239-aab5-0c5a254102a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv\" (UID: \"87c4642c-5571-4239-aab5-0c5a254102a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:10.164448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.164324 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g49\" (UniqueName: \"kubernetes.io/projected/87c4642c-5571-4239-aab5-0c5a254102a3-kube-api-access-74g49\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv\" (UID: \"87c4642c-5571-4239-aab5-0c5a254102a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:10.164924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.164902 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c4642c-5571-4239-aab5-0c5a254102a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv\" (UID: \"87c4642c-5571-4239-aab5-0c5a254102a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:10.177857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.177827 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" podUID="38c09e8f-cac7-465b-9cc6-ae4ed40212d8" containerName="payload-processing" containerID="cri-o://d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a" gracePeriod=25 Oct 30 14:40:10.196844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.196787 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g49\" (UniqueName: \"kubernetes.io/projected/87c4642c-5571-4239-aab5-0c5a254102a3-kube-api-access-74g49\") pod \"payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv\" (UID: \"87c4642c-5571-4239-aab5-0c5a254102a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:10.209695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.209662 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" podUID="5a692131-bb69-4e30-9069-8aa7c017d176" containerName="payload-pre-processing" containerID="cri-o://691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97" gracePeriod=25 Oct 30 14:40:10.236647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.235255 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:10.557547 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:40:10.557515 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c4642c_5571_4239_aab5_0c5a254102a3.slice/crio-f0b68acb1d72543a913a45fd5cadf006ec6b46b318643522ab41731d80647ca6 WatchSource:0}: Error finding container f0b68acb1d72543a913a45fd5cadf006ec6b46b318643522ab41731d80647ca6: Status 404 returned error can't find the container with id f0b68acb1d72543a913a45fd5cadf006ec6b46b318643522ab41731d80647ca6 Oct 30 14:40:10.558123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.558096 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv"] Oct 30 14:40:10.575023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.574996 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:40:10.595116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.595086 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:40:10.677011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.676979 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a692131-bb69-4e30-9069-8aa7c017d176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a692131-bb69-4e30-9069-8aa7c017d176-plugins-config-volume\") pod \"5a692131-bb69-4e30-9069-8aa7c017d176\" (UID: \"5a692131-bb69-4e30-9069-8aa7c017d176\") " Oct 30 14:40:10.677168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.677092 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a692131-bb69-4e30-9069-8aa7c017d176-kube-api-access-zc42z\" (UniqueName: \"kubernetes.io/projected/5a692131-bb69-4e30-9069-8aa7c017d176-kube-api-access-zc42z\") pod \"5a692131-bb69-4e30-9069-8aa7c017d176\" (UID: \"5a692131-bb69-4e30-9069-8aa7c017d176\") " Oct 30 14:40:10.677256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.677218 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-plugins-config-volume\") pod \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\" (UID: \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\") " Oct 30 14:40:10.677315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.677305 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-kube-api-access-2d2vm\" (UniqueName: \"kubernetes.io/projected/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-kube-api-access-2d2vm\") pod \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\" (UID: \"38c09e8f-cac7-465b-9cc6-ae4ed40212d8\") " Oct 30 14:40:10.678591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.678416 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a692131-bb69-4e30-9069-8aa7c017d176-plugins-config-volume" pod "5a692131-bb69-4e30-9069-8aa7c017d176" (UID: "5a692131-bb69-4e30-9069-8aa7c017d176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:10.678591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.678548 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-plugins-config-volume" pod "38c09e8f-cac7-465b-9cc6-ae4ed40212d8" (UID: "38c09e8f-cac7-465b-9cc6-ae4ed40212d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:10.680715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.680693 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a692131-bb69-4e30-9069-8aa7c017d176-kube-api-access-zc42z" pod "5a692131-bb69-4e30-9069-8aa7c017d176" (UID: "5a692131-bb69-4e30-9069-8aa7c017d176"). InnerVolumeSpecName "kube-api-access-zc42z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:10.683601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.683578 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-kube-api-access-2d2vm" pod "38c09e8f-cac7-465b-9cc6-ae4ed40212d8" (UID: "38c09e8f-cac7-465b-9cc6-ae4ed40212d8"). InnerVolumeSpecName "kube-api-access-2d2vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:10.779257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.779165 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d2vm\" (UniqueName: \"kubernetes.io/projected/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-kube-api-access-2d2vm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:10.779257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.779236 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a692131-bb69-4e30-9069-8aa7c017d176-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:10.779257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.779248 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc42z\" (UniqueName: \"kubernetes.io/projected/5a692131-bb69-4e30-9069-8aa7c017d176-kube-api-access-zc42z\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:10.779257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:10.779259 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c09e8f-cac7-465b-9cc6-ae4ed40212d8-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:11.137559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.137471 5824 generic.go:356] "Generic (PLEG): container finished" podID="38c09e8f-cac7-465b-9cc6-ae4ed40212d8" containerID="d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a" exitCode=0 Oct 30 14:40:11.137559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.137517 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" Oct 30 14:40:11.138082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.137554 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" event={"ID":"38c09e8f-cac7-465b-9cc6-ae4ed40212d8","Type":"ContainerDied","Data":"d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a"} Oct 30 14:40:11.138082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.137607 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf" event={"ID":"38c09e8f-cac7-465b-9cc6-ae4ed40212d8","Type":"ContainerDied","Data":"d9736590d01b13858f71123f35372132743e1e7b967ae33e183e5c6ace1901c8"} Oct 30 14:40:11.138082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.137641 5824 scope.go:122] "RemoveContainer" containerID="d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a" Oct 30 14:40:11.140147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.140118 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" event={"ID":"87c4642c-5571-4239-aab5-0c5a254102a3","Type":"ContainerStarted","Data":"910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f"} Oct 30 14:40:11.140296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.140183 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:11.140296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.140217 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" event={"ID":"87c4642c-5571-4239-aab5-0c5a254102a3","Type":"ContainerStarted","Data":"f0b68acb1d72543a913a45fd5cadf006ec6b46b318643522ab41731d80647ca6"} Oct 30 14:40:11.158878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.158842 5824 generic.go:356] "Generic (PLEG): container finished" podID="5a692131-bb69-4e30-9069-8aa7c017d176" containerID="691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97" exitCode=0 Oct 30 14:40:11.159290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.159264 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" event={"ID":"5a692131-bb69-4e30-9069-8aa7c017d176","Type":"ContainerDied","Data":"691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97"} Oct 30 14:40:11.159554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.159477 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" Oct 30 14:40:11.159554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.159530 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5" event={"ID":"5a692131-bb69-4e30-9069-8aa7c017d176","Type":"ContainerDied","Data":"71b04e20b3860cbb53e852e33a6f672aa7c39de9798db57ca4e698d399697f80"} Oct 30 14:40:11.160390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.160358 5824 scope.go:122] "RemoveContainer" containerID="d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a" Oct 30 14:40:11.160769 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:40:11.160737 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a\": container with ID starting with d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a not found: ID does not exist" containerID="d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a" Oct 30 14:40:11.160874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.160782 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a"} err="failed to get container status \"d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a\": rpc error: code = NotFound desc = could not find container \"d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a\": container with ID starting with d956d04be6cc01f94ee1f7fbebd07a888ed2d1ea9cb37e5271e41eb97d71e16a not found: ID does not exist" Oct 30 14:40:11.160874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.160827 5824 scope.go:122] "RemoveContainer" containerID="691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97" Oct 30 14:40:11.172688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.172671 5824 scope.go:122] "RemoveContainer" containerID="691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97" Oct 30 14:40:11.172986 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:40:11.172965 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97\": container with ID starting with 691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97 not found: ID does not exist" containerID="691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97" Oct 30 14:40:11.173039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.172993 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97"} err="failed to get container status \"691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97\": rpc error: code = NotFound desc = could not find container \"691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97\": container with ID starting with 691cbf09de4f3359888bed2c5877c377f680ea1eb4c8e567ad10f69abbe95c97 not found: ID does not exist" Oct 30 14:40:11.200423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.200398 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf"] Oct 30 14:40:11.212934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.212909 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0e8b9-7759fdbc48-f2lwf"] Oct 30 14:40:11.260367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.260311 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" podStartSLOduration=2.260296365 podStartE2EDuration="2.260296365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:11.257020543 +0000 UTC m=+3041.057525987" watchObservedRunningTime="2026-10-30 14:40:11.260296365 +0000 UTC m=+3041.060801809" Oct 30 14:40:11.301404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.301375 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5"] Oct 30 14:40:11.306440 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:11.306415 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0e8b9-74b8f4b587-gw4r5"] Oct 30 14:40:12.322349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:12.322317 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c09e8f-cac7-465b-9cc6-ae4ed40212d8" path="/var/lib/kubelet/pods/38c09e8f-cac7-465b-9cc6-ae4ed40212d8/volumes" Oct 30 14:40:12.322827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:12.322812 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a692131-bb69-4e30-9069-8aa7c017d176" path="/var/lib/kubelet/pods/5a692131-bb69-4e30-9069-8aa7c017d176/volumes" Oct 30 14:40:13.865827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.865795 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr"] Oct 30 14:40:13.866639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.866621 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5a692131-bb69-4e30-9069-8aa7c017d176" containerName="payload-pre-processing" Oct 30 14:40:13.866707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.866641 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a692131-bb69-4e30-9069-8aa7c017d176" containerName="payload-pre-processing" Oct 30 14:40:13.866707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.866663 5824 cpu_manager.go:414] "containerMap: removing container" podUID="38c09e8f-cac7-465b-9cc6-ae4ed40212d8" containerName="payload-processing" Oct 30 14:40:13.866707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.866668 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c09e8f-cac7-465b-9cc6-ae4ed40212d8" containerName="payload-processing" Oct 30 14:40:13.866810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.866804 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a692131-bb69-4e30-9069-8aa7c017d176" containerName="payload-pre-processing" Oct 30 14:40:13.866844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.866815 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c09e8f-cac7-465b-9cc6-ae4ed40212d8" containerName="payload-processing" Oct 30 14:40:13.872182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.872160 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:13.893719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.893692 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr"] Oct 30 14:40:13.902761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.902723 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:40:13.902916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:13.902807 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:40:14.027813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.027778 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4692t\" (UniqueName: \"kubernetes.io/projected/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-kube-api-access-4692t\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.028006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.027924 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.028125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.028103 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.129687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.129589 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4692t\" (UniqueName: \"kubernetes.io/projected/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-kube-api-access-4692t\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.129851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.129690 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.129851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.129806 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.130479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.130458 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.132282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.132255 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.156110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.156082 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4692t\" (UniqueName: \"kubernetes.io/projected/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-kube-api-access-4692t\") pod \"maas-api-revoke-sub-46035f76-mg7hr\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.186766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.186737 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:14.505082 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:40:14.505052 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92ec28c_f4b8_4bcb_9fd4_df80e87dfd42.slice/crio-e8489c62cf2aa141acc3e1b408ebab5737da97484bad914da8423777bfaaa593 WatchSource:0}: Error finding container e8489c62cf2aa141acc3e1b408ebab5737da97484bad914da8423777bfaaa593: Status 404 returned error can't find the container with id e8489c62cf2aa141acc3e1b408ebab5737da97484bad914da8423777bfaaa593 Oct 30 14:40:14.505548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:14.505522 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr"] Oct 30 14:40:15.193721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:15.193683 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" event={"ID":"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42","Type":"ContainerStarted","Data":"83b94125a4f886b261ebe4c02fb0816e2c76031265c5edda0e2a8f25d5a3f9e4"} Oct 30 14:40:15.194247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:15.193744 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" event={"ID":"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42","Type":"ContainerStarted","Data":"e8489c62cf2aa141acc3e1b408ebab5737da97484bad914da8423777bfaaa593"} Oct 30 14:40:15.237395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:15.237337 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" podStartSLOduration=2.237320742 podStartE2EDuration="2.237320742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:15.236735691 +0000 UTC m=+3045.037241136" watchObservedRunningTime="2026-10-30 14:40:15.237320742 +0000 UTC m=+3045.037826189" Oct 30 14:40:16.202382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:16.202341 5824 generic.go:356] "Generic (PLEG): container finished" podID="a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42" containerID="83b94125a4f886b261ebe4c02fb0816e2c76031265c5edda0e2a8f25d5a3f9e4" exitCode=0 Oct 30 14:40:16.202879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:16.202407 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" event={"ID":"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42","Type":"ContainerDied","Data":"83b94125a4f886b261ebe4c02fb0816e2c76031265c5edda0e2a8f25d5a3f9e4"} Oct 30 14:40:17.085852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.085818 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:40:17.383075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.383047 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:17.480800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.480761 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-kube-api-access-4692t\" (UniqueName: \"kubernetes.io/projected/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-kube-api-access-4692t\") pod \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " Oct 30 14:40:17.481016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.480838 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-openshift-ca\") pod \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " Oct 30 14:40:17.481016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.480928 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-xks-ca\") pod \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\" (UID: \"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42\") " Oct 30 14:40:17.482910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.482876 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-openshift-ca" pod "a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42" (UID: "a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:17.485232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.485180 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-kube-api-access-4692t" pod "a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42" (UID: "a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42"). InnerVolumeSpecName "kube-api-access-4692t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:17.501225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.486038 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-xks-ca" pod "a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42" (UID: "a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:40:17.583984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.583942 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4692t\" (UniqueName: \"kubernetes.io/projected/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-kube-api-access-4692t\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:17.583984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.583981 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:17.584208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:17.583998 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:18.220860 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:18.220826 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" Oct 30 14:40:18.221043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:18.220859 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr" event={"ID":"a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42","Type":"ContainerDied","Data":"e8489c62cf2aa141acc3e1b408ebab5737da97484bad914da8423777bfaaa593"} Oct 30 14:40:18.221043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:18.220903 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8489c62cf2aa141acc3e1b408ebab5737da97484bad914da8423777bfaaa593" Oct 30 14:40:22.171871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:22.171842 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:40:22.984143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:22.984107 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q"] Oct 30 14:40:22.984998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:22.984979 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42" containerName="revoke-keys" Oct 30 14:40:22.985051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:22.985001 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42" containerName="revoke-keys" Oct 30 14:40:22.985157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:22.985147 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42" containerName="revoke-keys" Oct 30 14:40:22.989400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:22.989378 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.015762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.015729 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q"] Oct 30 14:40:23.020411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.020360 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0efe43-openshift-default-dockercfg-bthxx\"" Oct 30 14:40:23.062663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.062626 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.062831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.062695 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.062831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.062756 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.062831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.062797 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd201164-e0c1-4544-9892-4353f4bfadcb-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.062948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.062896 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.062948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.062926 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.063015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.062991 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.063050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.063035 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.063127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.063107 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7sx5\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-kube-api-access-k7sx5\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.164650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.164612 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.164650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.164652 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.164906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.164690 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.164906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.164714 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.164906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.164772 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7sx5\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-kube-api-access-k7sx5\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.165069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.164969 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.165069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.165041 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.165181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.165077 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.165181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.165172 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd201164-e0c1-4544-9892-4353f4bfadcb-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.165349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.165264 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-credential-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.165408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.165350 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-socket\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.165478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.165444 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-data\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.165478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.165468 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-certs\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.166064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.166039 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd201164-e0c1-4544-9892-4353f4bfadcb-istiod-ca-cert\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.167287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.167258 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-envoy\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.167488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.167472 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-podinfo\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.194994 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.194967 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-token\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.195366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.195026 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7sx5\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-kube-api-access-k7sx5\") pod \"e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.304555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.304469 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:23.631624 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:40:23.631592 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd201164_e0c1_4544_9892_4353f4bfadcb.slice/crio-ead3c298b065f30f4f0e28f63fba6fa9470994d92ac93b3ce85a456b6eab5f3c WatchSource:0}: Error finding container ead3c298b065f30f4f0e28f63fba6fa9470994d92ac93b3ce85a456b6eab5f3c: Status 404 returned error can't find the container with id ead3c298b065f30f4f0e28f63fba6fa9470994d92ac93b3ce85a456b6eab5f3c Oct 30 14:40:23.632941 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.632915 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q"] Oct 30 14:40:23.634694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.634657 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:40:23.634791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.634727 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:40:23.634791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:23.634757 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:40:24.275172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.275126 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" event={"ID":"dd201164-e0c1-4544-9892-4353f4bfadcb","Type":"ContainerStarted","Data":"f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4"} Oct 30 14:40:24.275710 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.275232 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" event={"ID":"dd201164-e0c1-4544-9892-4353f4bfadcb","Type":"ContainerStarted","Data":"ead3c298b065f30f4f0e28f63fba6fa9470994d92ac93b3ce85a456b6eab5f3c"} Oct 30 14:40:24.306214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.305466 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:24.307475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.307240 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 14:40:24.307475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.307305 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 14:40:24.320959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.320928 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg"] Oct 30 14:40:24.325859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.325837 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.335483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.335177 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" podStartSLOduration=2.335162805 podStartE2EDuration="2.335162805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:24.33115069 +0000 UTC m=+3054.131656135" watchObservedRunningTime="2026-10-30 14:40:24.335162805 +0000 UTC m=+3054.135668250" Oct 30 14:40:24.356073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.354815 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg"] Oct 30 14:40:24.356669 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.356643 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:40:24.356871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.356857 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:40:24.403131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.403080 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.403657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.403631 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25fxq\" (UniqueName: \"kubernetes.io/projected/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-kube-api-access-25fxq\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.403776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.403666 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.505244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.505178 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fxq\" (UniqueName: \"kubernetes.io/projected/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-kube-api-access-25fxq\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.505442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.505252 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.505442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.505300 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.506022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.505999 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.507797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.507768 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.532816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.532754 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fxq\" (UniqueName: \"kubernetes.io/projected/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-kube-api-access-25fxq\") pod \"maas-api-revoke-sub-c547a94e-v6hzg\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.666629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.666586 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:24.984293 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:40:24.984255 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9a443b_f8c2_4fa6_89fd_06d43f9e5428.slice/crio-16c105c4e042cd38e8bfa3842be78d9468579a2e07f664513bc3f988b1858eab WatchSource:0}: Error finding container 16c105c4e042cd38e8bfa3842be78d9468579a2e07f664513bc3f988b1858eab: Status 404 returned error can't find the container with id 16c105c4e042cd38e8bfa3842be78d9468579a2e07f664513bc3f988b1858eab Oct 30 14:40:24.985753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:24.985727 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg"] Oct 30 14:40:25.284267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:25.284230 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" event={"ID":"db9a443b-f8c2-4fa6-89fd-06d43f9e5428","Type":"ContainerStarted","Data":"16c105c4e042cd38e8bfa3842be78d9468579a2e07f664513bc3f988b1858eab"} Oct 30 14:40:25.305570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:25.305531 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 14:40:25.305741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:25.305582 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 14:40:25.773943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:25.773900 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg"] Oct 30 14:40:25.803547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:25.803515 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg"] Oct 30 14:40:26.294090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:26.294053 5824 generic.go:356] "Generic (PLEG): container finished" podID="db9a443b-f8c2-4fa6-89fd-06d43f9e5428" containerID="a8f71f18891f49cc5ff91964ddc02b68ab3eb7db55e6fbe7ef8d4208011897be" exitCode=0 Oct 30 14:40:26.294575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:26.294151 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" event={"ID":"db9a443b-f8c2-4fa6-89fd-06d43f9e5428","Type":"ContainerDied","Data":"a8f71f18891f49cc5ff91964ddc02b68ab3eb7db55e6fbe7ef8d4208011897be"} Oct 30 14:40:26.305023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:26.304998 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 14:40:26.305114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:26.305044 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 14:40:27.305225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.305183 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 14:40:27.305513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.305249 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 14:40:27.508988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.508962 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:27.691271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.691243 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-xks-ca\") pod \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " Oct 30 14:40:27.691467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.691287 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-kube-api-access-25fxq\" (UniqueName: \"kubernetes.io/projected/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-kube-api-access-25fxq\") pod \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " Oct 30 14:40:27.691467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.691342 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-openshift-ca\") pod \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\" (UID: \"db9a443b-f8c2-4fa6-89fd-06d43f9e5428\") " Oct 30 14:40:27.691736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.691710 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-openshift-ca" pod "db9a443b-f8c2-4fa6-89fd-06d43f9e5428" (UID: "db9a443b-f8c2-4fa6-89fd-06d43f9e5428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:27.692075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.692056 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:27.693577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.693547 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-kube-api-access-25fxq" pod "db9a443b-f8c2-4fa6-89fd-06d43f9e5428" (UID: "db9a443b-f8c2-4fa6-89fd-06d43f9e5428"). InnerVolumeSpecName "kube-api-access-25fxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:27.693664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.693578 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-xks-ca" pod "db9a443b-f8c2-4fa6-89fd-06d43f9e5428" (UID: "db9a443b-f8c2-4fa6-89fd-06d43f9e5428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:40:27.793736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.793702 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:27.793736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:27.793732 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25fxq\" (UniqueName: \"kubernetes.io/projected/db9a443b-f8c2-4fa6-89fd-06d43f9e5428-kube-api-access-25fxq\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:28.304863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:28.304827 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 14:40:28.305084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:28.304884 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 14:40:28.314294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:28.314264 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" Oct 30 14:40:28.320414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:28.320386 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg" event={"ID":"db9a443b-f8c2-4fa6-89fd-06d43f9e5428","Type":"ContainerDied","Data":"16c105c4e042cd38e8bfa3842be78d9468579a2e07f664513bc3f988b1858eab"} Oct 30 14:40:28.320558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:28.320440 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c105c4e042cd38e8bfa3842be78d9468579a2e07f664513bc3f988b1858eab" Oct 30 14:40:29.305525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:29.305489 5824 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 14:40:29.305743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:29.305541 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 14:40:30.336826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:30.336784 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:30.337450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:30.337417 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:30.338525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:30.338502 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:40:30.774549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:30.774489 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" podUID="fad4515b-90dd-4418-83e6-879fed176399" containerName="payload-processing" containerID="cri-o://f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2" gracePeriod=25 Oct 30 14:40:30.804735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:30.804704 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" podUID="cdf724df-3ede-412e-814d-5feb227da664" containerName="payload-pre-processing" containerID="cri-o://b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f" gracePeriod=25 Oct 30 14:40:31.124117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.124092 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:40:31.127991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.127968 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:40:31.141744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.141718 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad4515b-90dd-4418-83e6-879fed176399-kube-api-access-hbmd7\" (UniqueName: \"kubernetes.io/projected/fad4515b-90dd-4418-83e6-879fed176399-kube-api-access-hbmd7\") pod \"fad4515b-90dd-4418-83e6-879fed176399\" (UID: \"fad4515b-90dd-4418-83e6-879fed176399\") " Oct 30 14:40:31.141870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.141801 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf724df-3ede-412e-814d-5feb227da664-kube-api-access-2sbrz\" (UniqueName: \"kubernetes.io/projected/cdf724df-3ede-412e-814d-5feb227da664-kube-api-access-2sbrz\") pod \"cdf724df-3ede-412e-814d-5feb227da664\" (UID: \"cdf724df-3ede-412e-814d-5feb227da664\") " Oct 30 14:40:31.141870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.141828 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf724df-3ede-412e-814d-5feb227da664-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf724df-3ede-412e-814d-5feb227da664-plugins-config-volume\") pod \"cdf724df-3ede-412e-814d-5feb227da664\" (UID: \"cdf724df-3ede-412e-814d-5feb227da664\") " Oct 30 14:40:31.141963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.141936 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad4515b-90dd-4418-83e6-879fed176399-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4515b-90dd-4418-83e6-879fed176399-plugins-config-volume\") pod \"fad4515b-90dd-4418-83e6-879fed176399\" (UID: \"fad4515b-90dd-4418-83e6-879fed176399\") " Oct 30 14:40:31.142319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.142292 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf724df-3ede-412e-814d-5feb227da664-plugins-config-volume" pod "cdf724df-3ede-412e-814d-5feb227da664" (UID: "cdf724df-3ede-412e-814d-5feb227da664"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:31.142425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.142388 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad4515b-90dd-4418-83e6-879fed176399-plugins-config-volume" pod "fad4515b-90dd-4418-83e6-879fed176399" (UID: "fad4515b-90dd-4418-83e6-879fed176399"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:40:31.142686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.142668 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf724df-3ede-412e-814d-5feb227da664-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:31.142760 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.142687 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4515b-90dd-4418-83e6-879fed176399-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:31.143915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.143893 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad4515b-90dd-4418-83e6-879fed176399-kube-api-access-hbmd7" pod "fad4515b-90dd-4418-83e6-879fed176399" (UID: "fad4515b-90dd-4418-83e6-879fed176399"). InnerVolumeSpecName "kube-api-access-hbmd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:31.143995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.143911 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf724df-3ede-412e-814d-5feb227da664-kube-api-access-2sbrz" pod "cdf724df-3ede-412e-814d-5feb227da664" (UID: "cdf724df-3ede-412e-814d-5feb227da664"). InnerVolumeSpecName "kube-api-access-2sbrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:40:31.245969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.245928 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbmd7\" (UniqueName: \"kubernetes.io/projected/fad4515b-90dd-4418-83e6-879fed176399-kube-api-access-hbmd7\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:31.245969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.245971 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sbrz\" (UniqueName: \"kubernetes.io/projected/cdf724df-3ede-412e-814d-5feb227da664-kube-api-access-2sbrz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:40:31.340481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.340397 5824 generic.go:356] "Generic (PLEG): container finished" podID="fad4515b-90dd-4418-83e6-879fed176399" containerID="f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2" exitCode=0 Oct 30 14:40:31.340481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.340447 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" Oct 30 14:40:31.340481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.340473 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" event={"ID":"fad4515b-90dd-4418-83e6-879fed176399","Type":"ContainerDied","Data":"f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2"} Oct 30 14:40:31.341018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.340524 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg" event={"ID":"fad4515b-90dd-4418-83e6-879fed176399","Type":"ContainerDied","Data":"9cc1375b7b299290a7b1776825fc1b333d6d9ec75755f222d95e4962dd16145e"} Oct 30 14:40:31.341018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.340546 5824 scope.go:122] "RemoveContainer" containerID="f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2" Oct 30 14:40:31.342171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.342148 5824 generic.go:356] "Generic (PLEG): container finished" podID="cdf724df-3ede-412e-814d-5feb227da664" containerID="b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f" exitCode=0 Oct 30 14:40:31.342380 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.342354 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" event={"ID":"cdf724df-3ede-412e-814d-5feb227da664","Type":"ContainerDied","Data":"b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f"} Oct 30 14:40:31.342460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.342410 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" event={"ID":"cdf724df-3ede-412e-814d-5feb227da664","Type":"ContainerDied","Data":"51e08e82e9d79ab4c3ed257d33e9077f727c8676a7a85d15d9296749e3078849"} Oct 30 14:40:31.342460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.342428 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg" Oct 30 14:40:31.352345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.352323 5824 scope.go:122] "RemoveContainer" containerID="f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2" Oct 30 14:40:31.352630 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:40:31.352607 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2\": container with ID starting with f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2 not found: ID does not exist" containerID="f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2" Oct 30 14:40:31.352696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.352641 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2"} err="failed to get container status \"f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2\": rpc error: code = NotFound desc = could not find container \"f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2\": container with ID starting with f4a8dd82100d93dc7771b092544f9b33e1943a5b36c0bf58bdddf21983201fe2 not found: ID does not exist" Oct 30 14:40:31.352696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.352662 5824 scope.go:122] "RemoveContainer" containerID="b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f" Oct 30 14:40:31.362445 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.362428 5824 scope.go:122] "RemoveContainer" containerID="b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f" Oct 30 14:40:31.362705 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:40:31.362685 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f\": container with ID starting with b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f not found: ID does not exist" containerID="b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f" Oct 30 14:40:31.362756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.362712 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f"} err="failed to get container status \"b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f\": rpc error: code = NotFound desc = could not find container \"b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f\": container with ID starting with b13b86618fbbad8e71b68b186e22f5f646fb0846a983f96ee1f45e33e77d691f not found: ID does not exist" Oct 30 14:40:31.403046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.403008 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg"] Oct 30 14:40:31.406605 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.406580 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-460b0b-86df895954-nvbqg"] Oct 30 14:40:31.455316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.455284 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg"] Oct 30 14:40:31.471390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:31.471360 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-460b0b-77fcc4cc9-ct6xg"] Oct 30 14:40:32.319358 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.319325 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf724df-3ede-412e-814d-5feb227da664" path="/var/lib/kubelet/pods/cdf724df-3ede-412e-814d-5feb227da664/volumes" Oct 30 14:40:32.319796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.319781 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad4515b-90dd-4418-83e6-879fed176399" path="/var/lib/kubelet/pods/fad4515b-90dd-4418-83e6-879fed176399/volumes" Oct 30 14:40:32.928065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.928031 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg"] Oct 30 14:40:32.929212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929169 5824 cpu_manager.go:414] "containerMap: removing container" podUID="cdf724df-3ede-412e-814d-5feb227da664" containerName="payload-pre-processing" Oct 30 14:40:32.929212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929202 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf724df-3ede-412e-814d-5feb227da664" containerName="payload-pre-processing" Oct 30 14:40:32.929322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929220 5824 cpu_manager.go:414] "containerMap: removing container" podUID="db9a443b-f8c2-4fa6-89fd-06d43f9e5428" containerName="revoke-keys" Oct 30 14:40:32.929322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929228 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9a443b-f8c2-4fa6-89fd-06d43f9e5428" containerName="revoke-keys" Oct 30 14:40:32.929322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929263 5824 cpu_manager.go:414] "containerMap: removing container" podUID="fad4515b-90dd-4418-83e6-879fed176399" containerName="payload-processing" Oct 30 14:40:32.929322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929271 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad4515b-90dd-4418-83e6-879fed176399" containerName="payload-processing" Oct 30 14:40:32.929462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929450 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf724df-3ede-412e-814d-5feb227da664" containerName="payload-pre-processing" Oct 30 14:40:32.929506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929468 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9a443b-f8c2-4fa6-89fd-06d43f9e5428" containerName="revoke-keys" Oct 30 14:40:32.929506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.929481 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad4515b-90dd-4418-83e6-879fed176399" containerName="payload-processing" Oct 30 14:40:32.935285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.935266 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:32.957006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.956976 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg"] Oct 30 14:40:32.966518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.966491 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 14:40:32.966724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.966491 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 14:40:32.966724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.966509 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 14:40:32.966919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.966625 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zzhpw\"" Oct 30 14:40:32.970316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.970291 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqrq4\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-kube-api-access-bqrq4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:32.970423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.970328 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:32.970590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.970573 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:32.970705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.970642 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:32.970766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.970719 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:32.970826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.970764 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:32.970898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:32.970880 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.071949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.071914 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.071949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.071958 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.072243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.072076 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.072243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.072180 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.072371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.072242 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqrq4\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-kube-api-access-bqrq4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.072371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.072272 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.072540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.072523 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.073030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.073006 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.074626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.074597 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.074723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.074645 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.074723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.074707 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.074953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.074932 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.101136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.101101 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.101290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.101226 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqrq4\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-kube-api-access-bqrq4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-68bcg\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.249306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.249175 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:33.583003 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:40:33.582973 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d5e1f9_6e09_4021_a6f7_9a3546efb0c4.slice/crio-ac8301e406212f1fd54f024e9fbf7e081366492ba95a1440f1b0353ac253b37e WatchSource:0}: Error finding container ac8301e406212f1fd54f024e9fbf7e081366492ba95a1440f1b0353ac253b37e: Status 404 returned error can't find the container with id ac8301e406212f1fd54f024e9fbf7e081366492ba95a1440f1b0353ac253b37e Oct 30 14:40:33.584412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.584389 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg"] Oct 30 14:40:33.586277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.586243 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:40:33.586349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:33.586304 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:40:33.646447 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:40:33.646414 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:40:34.369964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:34.369921 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" event={"ID":"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4","Type":"ContainerStarted","Data":"1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf"} Oct 30 14:40:34.370489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:34.369987 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" event={"ID":"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4","Type":"ContainerStarted","Data":"ac8301e406212f1fd54f024e9fbf7e081366492ba95a1440f1b0353ac253b37e"} Oct 30 14:40:34.370489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:34.370089 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:34.372714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:34.372683 5824 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-68bcg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:40:34.372883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:34.372758 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" podUID="97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:40:35.379560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:35.379526 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:40:35.465215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:40:35.464335 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" podStartSLOduration=3.4643134939999998 podStartE2EDuration="3.464313494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:40:34.451571194 +0000 UTC m=+3064.252076638" watchObservedRunningTime="2026-10-30 14:40:35.464313494 +0000 UTC m=+3065.264818941" Oct 30 14:41:03.646985 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:41:03.646945 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:41:05.056991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:05.056954 5824 patch_prober.go:29] interesting pod/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": read tcp 10.134.0.2:41146->10.134.0.199:15021: read: connection reset by peer" start-of-body= Oct 30 14:41:05.057430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:05.057034 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": read tcp 10.134.0.2:41146->10.134.0.199:15021: read: connection reset by peer" Oct 30 14:41:05.743408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:05.743372 5824 generic.go:356] "Generic (PLEG): container finished" podID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerID="a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c" exitCode=0 Oct 30 14:41:05.743632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:05.743476 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" event={"ID":"06f88aaf-4caa-47d3-badd-9a26b264052b","Type":"ContainerDied","Data":"a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c"} Oct 30 14:41:05.743960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:05.743945 5824 scope.go:122] "RemoveContainer" containerID="a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c" Oct 30 14:41:05.745925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:05.745890 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:41:05.746044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:05.745987 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:41:05.746118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:05.746040 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:41:06.795255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:06.795217 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" event={"ID":"06f88aaf-4caa-47d3-badd-9a26b264052b","Type":"ContainerStarted","Data":"b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2"} Oct 30 14:41:07.624540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:07.624505 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:41:07.624725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:07.624662 5824 patch_prober.go:29] interesting pod/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 14:41:07.624772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:07.624734 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 14:41:08.624873 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:08.624832 5824 patch_prober.go:29] interesting pod/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 14:41:08.625338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:08.624905 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 14:41:09.624019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:09.623985 5824 patch_prober.go:29] interesting pod/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 14:41:09.624225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:09.624041 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 14:41:10.624763 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:10.624725 5824 patch_prober.go:29] interesting pod/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 14:41:10.625166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:10.624803 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 14:41:11.624236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:11.624205 5824 patch_prober.go:29] interesting pod/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 30 14:41:11.624411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:11.624267 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 30 14:41:12.629152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:12.629122 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:41:12.629615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:12.629592 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:41:12.630454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:12.630435 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:41:32.121363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:32.121332 5824 scope.go:122] "RemoveContainer" containerID="14d45ad4823fac09c75c0590651f1fc6883c3f75d210fec6bfdf744b5449ff45" Oct 30 14:41:32.131960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:32.131939 5824 scope.go:122] "RemoveContainer" containerID="3629632fd2861982e5c09106bf64439e60d1d57224d4a60b8efd9ee9c23fbe2d" Oct 30 14:41:33.643697 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:41:33.643657 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:41:50.054175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.054138 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg"] Oct 30 14:41:50.059458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.059433 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.079434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.079403 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg"] Oct 30 14:41:50.089071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.089020 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:41:50.089634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.089127 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:41:50.169975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.169931 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.170160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.170117 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4xj\" (UniqueName: \"kubernetes.io/projected/06fd4a14-1587-4b2a-b925-082a144415c2-kube-api-access-zw4xj\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.170288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.170270 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.271561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.271526 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.271793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.271751 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.271945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.271892 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4xj\" (UniqueName: \"kubernetes.io/projected/06fd4a14-1587-4b2a-b925-082a144415c2-kube-api-access-zw4xj\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.272444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.272419 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.273983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.273961 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.300118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.300085 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4xj\" (UniqueName: \"kubernetes.io/projected/06fd4a14-1587-4b2a-b925-082a144415c2-kube-api-access-zw4xj\") pod \"maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.379079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.378985 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:50.689419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.689375 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg"] Oct 30 14:41:50.689692 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:41:50.689670 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fd4a14_1587_4b2a_b925_082a144415c2.slice/crio-8b20b6df9bba98a62815e1ff9e854d613eee8e5d64ae9ea0f38c666dc566c9f1 WatchSource:0}: Error finding container 8b20b6df9bba98a62815e1ff9e854d613eee8e5d64ae9ea0f38c666dc566c9f1: Status 404 returned error can't find the container with id 8b20b6df9bba98a62815e1ff9e854d613eee8e5d64ae9ea0f38c666dc566c9f1 Oct 30 14:41:50.985258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.985226 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d"] Oct 30 14:41:50.989975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:50.989954 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.026119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.026072 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2927fd-serving-cert\"" Oct 30 14:41:51.031345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.031315 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d"] Oct 30 14:41:51.084818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.084792 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.085225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.084851 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-metrics-certs\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.085225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.084964 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.085225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.085164 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dng\" (UniqueName: \"kubernetes.io/projected/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-kube-api-access-w8dng\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.186276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.186230 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.186466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.186426 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-metrics-certs\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.186543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.186521 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.186727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.186709 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dng\" (UniqueName: \"kubernetes.io/projected/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-kube-api-access-w8dng\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.187076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.187045 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.187232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.187185 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" event={"ID":"06fd4a14-1587-4b2a-b925-082a144415c2","Type":"ContainerStarted","Data":"75db56c6bb3e02c12002212e99a10f2eefed33f8adfef1d1101bb10d9ad96207"} Oct 30 14:41:51.187303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.187287 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" event={"ID":"06fd4a14-1587-4b2a-b925-082a144415c2","Type":"ContainerStarted","Data":"8b20b6df9bba98a62815e1ff9e854d613eee8e5d64ae9ea0f38c666dc566c9f1"} Oct 30 14:41:51.188996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.188975 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-metrics-certs\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.189395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.189371 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.215303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.215220 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dng\" (UniqueName: \"kubernetes.io/projected/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-kube-api-access-w8dng\") pod \"maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.246081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.246010 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" podStartSLOduration=1.245989246 podStartE2EDuration="1.245989246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:51.24050515 +0000 UTC m=+3141.041010591" watchObservedRunningTime="2026-10-30 14:41:51.245989246 +0000 UTC m=+3141.046494692" Oct 30 14:41:51.303560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.303490 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:51.626417 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:41:51.626388 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3892604c_04ee_4ce3_b7a1_f4cbf6cfb99a.slice/crio-a7061ad4abe34bfd68aac97462baf35131ccfbb3ef22cb6194eee61e65e1c72b WatchSource:0}: Error finding container a7061ad4abe34bfd68aac97462baf35131ccfbb3ef22cb6194eee61e65e1c72b: Status 404 returned error can't find the container with id a7061ad4abe34bfd68aac97462baf35131ccfbb3ef22cb6194eee61e65e1c72b Oct 30 14:41:51.627849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:51.627821 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d"] Oct 30 14:41:52.196919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:52.196879 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" event={"ID":"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a","Type":"ContainerStarted","Data":"fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0"} Oct 30 14:41:52.197448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:52.196946 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" event={"ID":"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a","Type":"ContainerStarted","Data":"a7061ad4abe34bfd68aac97462baf35131ccfbb3ef22cb6194eee61e65e1c72b"} Oct 30 14:41:52.197448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:52.196974 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:41:52.198842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:52.198810 5824 generic.go:356] "Generic (PLEG): container finished" podID="06fd4a14-1587-4b2a-b925-082a144415c2" containerID="75db56c6bb3e02c12002212e99a10f2eefed33f8adfef1d1101bb10d9ad96207" exitCode=0 Oct 30 14:41:52.198994 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:52.198890 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" event={"ID":"06fd4a14-1587-4b2a-b925-082a144415c2","Type":"ContainerDied","Data":"75db56c6bb3e02c12002212e99a10f2eefed33f8adfef1d1101bb10d9ad96207"} Oct 30 14:41:52.247506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:52.247422 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" podStartSLOduration=1.879262124 podStartE2EDuration="2.247409885s" totalImagesPullingTime="368.147761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:41:52.243237173 +0000 UTC m=+3142.043742610" watchObservedRunningTime="2026-10-30 14:41:52.247409885 +0000 UTC m=+3142.047915371" Oct 30 14:41:53.376974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.376948 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:53.523822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.523662 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fd4a14-1587-4b2a-b925-082a144415c2-kube-api-access-zw4xj\" (UniqueName: \"kubernetes.io/projected/06fd4a14-1587-4b2a-b925-082a144415c2-kube-api-access-zw4xj\") pod \"06fd4a14-1587-4b2a-b925-082a144415c2\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " Oct 30 14:41:53.523822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.523796 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-xks-ca\") pod \"06fd4a14-1587-4b2a-b925-082a144415c2\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " Oct 30 14:41:53.524073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.523971 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-openshift-ca\") pod \"06fd4a14-1587-4b2a-b925-082a144415c2\" (UID: \"06fd4a14-1587-4b2a-b925-082a144415c2\") " Oct 30 14:41:53.533614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.528134 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-openshift-ca" pod "06fd4a14-1587-4b2a-b925-082a144415c2" (UID: "06fd4a14-1587-4b2a-b925-082a144415c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:41:53.533614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.532653 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fd4a14-1587-4b2a-b925-082a144415c2-kube-api-access-zw4xj" pod "06fd4a14-1587-4b2a-b925-082a144415c2" (UID: "06fd4a14-1587-4b2a-b925-082a144415c2"). InnerVolumeSpecName "kube-api-access-zw4xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:41:53.546603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.545896 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-xks-ca" pod "06fd4a14-1587-4b2a-b925-082a144415c2" (UID: "06fd4a14-1587-4b2a-b925-082a144415c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:41:53.625695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.625658 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:41:53.625695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.625691 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw4xj\" (UniqueName: \"kubernetes.io/projected/06fd4a14-1587-4b2a-b925-082a144415c2-kube-api-access-zw4xj\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:41:53.625695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:53.625705 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fd4a14-1587-4b2a-b925-082a144415c2-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:41:54.216106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:54.216070 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" Oct 30 14:41:54.216277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:54.216071 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg" event={"ID":"06fd4a14-1587-4b2a-b925-082a144415c2","Type":"ContainerDied","Data":"8b20b6df9bba98a62815e1ff9e854d613eee8e5d64ae9ea0f38c666dc566c9f1"} Oct 30 14:41:54.216277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:54.216205 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b20b6df9bba98a62815e1ff9e854d613eee8e5d64ae9ea0f38c666dc566c9f1" Oct 30 14:41:58.214249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:41:58.214212 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:42:03.645467 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:42:03.645425 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:42:05.960135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:05.960098 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679"] Oct 30 14:42:06.894224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:06.894175 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn"] Oct 30 14:42:06.895019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:06.895003 5824 cpu_manager.go:414] "containerMap: removing container" podUID="06fd4a14-1587-4b2a-b925-082a144415c2" containerName="revoke-keys" Oct 30 14:42:06.895088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:06.895020 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fd4a14-1587-4b2a-b925-082a144415c2" containerName="revoke-keys" Oct 30 14:42:06.895155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:06.895144 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fd4a14-1587-4b2a-b925-082a144415c2" containerName="revoke-keys" Oct 30 14:42:06.897980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:06.897960 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:06.939137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:06.936864 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn"] Oct 30 14:42:06.959101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:06.959060 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:42:06.959349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:06.959324 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:42:07.041604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.041565 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.042022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.041805 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.042022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.041856 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpn9z\" (UniqueName: \"kubernetes.io/projected/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-kube-api-access-rpn9z\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.143554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.143518 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.143736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.143681 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.143736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.143717 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpn9z\" (UniqueName: \"kubernetes.io/projected/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-kube-api-access-rpn9z\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.144155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.144128 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.146089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.146033 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.169751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.169720 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpn9z\" (UniqueName: \"kubernetes.io/projected/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-kube-api-access-rpn9z\") pod \"maas-api-revoke-sub-be3003b5-jt2pn\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.260839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.260808 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:07.573010 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:42:07.572973 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d817e56_c250_4ecc_9865_b2c7e2a0d72d.slice/crio-5ea6ec34366a1320e14eb609a9fa2c56500495b059c7942717fb7a94330e73ef WatchSource:0}: Error finding container 5ea6ec34366a1320e14eb609a9fa2c56500495b059c7942717fb7a94330e73ef: Status 404 returned error can't find the container with id 5ea6ec34366a1320e14eb609a9fa2c56500495b059c7942717fb7a94330e73ef Oct 30 14:42:07.573662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:07.573631 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn"] Oct 30 14:42:08.347949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:08.347849 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" event={"ID":"5d817e56-c250-4ecc-9865-b2c7e2a0d72d","Type":"ContainerStarted","Data":"61bbd7bb8bbfe45d698c4f6ea398f2b036ee18bbc571413deaba7c90daa67066"} Oct 30 14:42:08.347949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:08.347920 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" event={"ID":"5d817e56-c250-4ecc-9865-b2c7e2a0d72d","Type":"ContainerStarted","Data":"5ea6ec34366a1320e14eb609a9fa2c56500495b059c7942717fb7a94330e73ef"} Oct 30 14:42:08.398822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:08.398764 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" podStartSLOduration=2.398750218 podStartE2EDuration="2.398750218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:42:08.394960369 +0000 UTC m=+3158.195465825" watchObservedRunningTime="2026-10-30 14:42:08.398750218 +0000 UTC m=+3158.199255661" Oct 30 14:42:09.356978 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:09.356942 5824 generic.go:356] "Generic (PLEG): container finished" podID="5d817e56-c250-4ecc-9865-b2c7e2a0d72d" containerID="61bbd7bb8bbfe45d698c4f6ea398f2b036ee18bbc571413deaba7c90daa67066" exitCode=0 Oct 30 14:42:09.357441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:09.357032 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" event={"ID":"5d817e56-c250-4ecc-9865-b2c7e2a0d72d","Type":"ContainerDied","Data":"61bbd7bb8bbfe45d698c4f6ea398f2b036ee18bbc571413deaba7c90daa67066"} Oct 30 14:42:10.584265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.584235 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:10.701128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.701096 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-xks-ca\") pod \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " Oct 30 14:42:10.701379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.701168 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-kube-api-access-rpn9z\" (UniqueName: \"kubernetes.io/projected/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-kube-api-access-rpn9z\") pod \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " Oct 30 14:42:10.701379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.701254 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-openshift-ca\") pod \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\" (UID: \"5d817e56-c250-4ecc-9865-b2c7e2a0d72d\") " Oct 30 14:42:10.701691 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.701658 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-openshift-ca" pod "5d817e56-c250-4ecc-9865-b2c7e2a0d72d" (UID: "5d817e56-c250-4ecc-9865-b2c7e2a0d72d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:42:10.702011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.701989 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:10.703908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.703876 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-xks-ca" pod "5d817e56-c250-4ecc-9865-b2c7e2a0d72d" (UID: "5d817e56-c250-4ecc-9865-b2c7e2a0d72d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:42:10.704242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.704174 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-kube-api-access-rpn9z" pod "5d817e56-c250-4ecc-9865-b2c7e2a0d72d" (UID: "5d817e56-c250-4ecc-9865-b2c7e2a0d72d"). InnerVolumeSpecName "kube-api-access-rpn9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:42:10.803899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.803864 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:10.803899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.803892 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpn9z\" (UniqueName: \"kubernetes.io/projected/5d817e56-c250-4ecc-9865-b2c7e2a0d72d-kube-api-access-rpn9z\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:10.961239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:10.961056 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" podUID="e88d8cdb-c0bf-4390-894e-ce91cfd1414b" containerName="payload-pre-processing" containerID="cri-o://bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5" gracePeriod=25 Oct 30 14:42:11.221554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.221525 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:42:11.311906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.311872 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-kube-api-access-knd46\" (UniqueName: \"kubernetes.io/projected/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-kube-api-access-knd46\") pod \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\" (UID: \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\") " Oct 30 14:42:11.312087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.311924 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-plugins-config-volume\") pod \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\" (UID: \"e88d8cdb-c0bf-4390-894e-ce91cfd1414b\") " Oct 30 14:42:11.312350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.312323 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-plugins-config-volume" pod "e88d8cdb-c0bf-4390-894e-ce91cfd1414b" (UID: "e88d8cdb-c0bf-4390-894e-ce91cfd1414b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:42:11.312748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.312725 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:11.314090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.314062 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-kube-api-access-knd46" pod "e88d8cdb-c0bf-4390-894e-ce91cfd1414b" (UID: "e88d8cdb-c0bf-4390-894e-ce91cfd1414b"). InnerVolumeSpecName "kube-api-access-knd46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:42:11.379132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.379054 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" Oct 30 14:42:11.379381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.379057 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn" event={"ID":"5d817e56-c250-4ecc-9865-b2c7e2a0d72d","Type":"ContainerDied","Data":"5ea6ec34366a1320e14eb609a9fa2c56500495b059c7942717fb7a94330e73ef"} Oct 30 14:42:11.379381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.379222 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea6ec34366a1320e14eb609a9fa2c56500495b059c7942717fb7a94330e73ef" Oct 30 14:42:11.381139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.381105 5824 generic.go:356] "Generic (PLEG): container finished" podID="e88d8cdb-c0bf-4390-894e-ce91cfd1414b" containerID="bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5" exitCode=0 Oct 30 14:42:11.381298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.381162 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" Oct 30 14:42:11.381298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.381183 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" event={"ID":"e88d8cdb-c0bf-4390-894e-ce91cfd1414b","Type":"ContainerDied","Data":"bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5"} Oct 30 14:42:11.381298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.381263 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679" event={"ID":"e88d8cdb-c0bf-4390-894e-ce91cfd1414b","Type":"ContainerDied","Data":"31dac9a2c830bd28b612da3e788ba316e60f47269cf20f1a9619988b7fbb6bb5"} Oct 30 14:42:11.381298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.381285 5824 scope.go:122] "RemoveContainer" containerID="bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5" Oct 30 14:42:11.393241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.393174 5824 scope.go:122] "RemoveContainer" containerID="bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5" Oct 30 14:42:11.393443 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:42:11.393427 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5\": container with ID starting with bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5 not found: ID does not exist" containerID="bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5" Oct 30 14:42:11.393504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.393450 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5"} err="failed to get container status \"bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5\": rpc error: code = NotFound desc = could not find container \"bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5\": container with ID starting with bbe336cef4769aca071500496f8b2c03b31533dcec20e4c07da64c6ec67fa5c5 not found: ID does not exist" Oct 30 14:42:11.413863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.413841 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knd46\" (UniqueName: \"kubernetes.io/projected/e88d8cdb-c0bf-4390-894e-ce91cfd1414b-kube-api-access-knd46\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:11.463839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.463764 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679"] Oct 30 14:42:11.467632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:11.467607 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a1854b-9d64cdc8c-dr679"] Oct 30 14:42:12.319256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:12.319220 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88d8cdb-c0bf-4390-894e-ce91cfd1414b" path="/var/lib/kubelet/pods/e88d8cdb-c0bf-4390-894e-ce91cfd1414b/volumes" Oct 30 14:42:15.544376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:15.544343 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv"] Oct 30 14:42:17.256772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.256738 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d"] Oct 30 14:42:17.257564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.257548 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e88d8cdb-c0bf-4390-894e-ce91cfd1414b" containerName="payload-pre-processing" Oct 30 14:42:17.257625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.257566 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88d8cdb-c0bf-4390-894e-ce91cfd1414b" containerName="payload-pre-processing" Oct 30 14:42:17.257625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.257600 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5d817e56-c250-4ecc-9865-b2c7e2a0d72d" containerName="revoke-keys" Oct 30 14:42:17.257625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.257606 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d817e56-c250-4ecc-9865-b2c7e2a0d72d" containerName="revoke-keys" Oct 30 14:42:17.257730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.257725 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d817e56-c250-4ecc-9865-b2c7e2a0d72d" containerName="revoke-keys" Oct 30 14:42:17.257764 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.257742 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88d8cdb-c0bf-4390-894e-ce91cfd1414b" containerName="payload-pre-processing" Oct 30 14:42:17.261781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.261763 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.283714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.283684 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d"] Oct 30 14:42:17.292696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.292665 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:42:17.292865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.292819 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:42:17.302874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.302829 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxc5\" (UniqueName: \"kubernetes.io/projected/f080aad9-b186-4a5e-be3a-597a046e836a-kube-api-access-ggxc5\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.303066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.302943 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.303236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.303213 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.404965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.404926 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.405167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.405018 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxc5\" (UniqueName: \"kubernetes.io/projected/f080aad9-b186-4a5e-be3a-597a046e836a-kube-api-access-ggxc5\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.405167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.405117 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.405820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.405799 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.407446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.407427 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.432070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.432040 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxc5\" (UniqueName: \"kubernetes.io/projected/f080aad9-b186-4a5e-be3a-597a046e836a-kube-api-access-ggxc5\") pod \"maas-api-revoke-sub-cd77c42d-7qz7d\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.587478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.587397 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:17.897989 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:42:17.897957 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf080aad9_b186_4a5e_be3a_597a046e836a.slice/crio-a1f02851c2ca455b1f80e63f4bfbc4a756437d58087d2762aba649f10245949f WatchSource:0}: Error finding container a1f02851c2ca455b1f80e63f4bfbc4a756437d58087d2762aba649f10245949f: Status 404 returned error can't find the container with id a1f02851c2ca455b1f80e63f4bfbc4a756437d58087d2762aba649f10245949f Oct 30 14:42:17.899050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:17.899013 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d"] Oct 30 14:42:18.438012 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:18.437979 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" event={"ID":"f080aad9-b186-4a5e-be3a-597a046e836a","Type":"ContainerStarted","Data":"18e5972ef39a2376561ded34314c9a02b31e57d9518b2fc0aa1ef3310c571196"} Oct 30 14:42:18.438395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:18.438028 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" event={"ID":"f080aad9-b186-4a5e-be3a-597a046e836a","Type":"ContainerStarted","Data":"a1f02851c2ca455b1f80e63f4bfbc4a756437d58087d2762aba649f10245949f"} Oct 30 14:42:18.485086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:18.485032 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" podStartSLOduration=1.4850179190000001 podStartE2EDuration="1.485017919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:42:18.482084958 +0000 UTC m=+3168.282590402" watchObservedRunningTime="2026-10-30 14:42:18.485017919 +0000 UTC m=+3168.285523363" Oct 30 14:42:19.324400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.324368 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z"] Oct 30 14:42:19.328184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.328166 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.348712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.348679 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z"] Oct 30 14:42:19.359362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.359316 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3b95dd-serving-cert\"" Oct 30 14:42:19.434521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.434482 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkt7\" (UniqueName: \"kubernetes.io/projected/c3eebbff-a656-4bc8-a369-3da7d76830f7-kube-api-access-nbkt7\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.434729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.434650 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3eebbff-a656-4bc8-a369-3da7d76830f7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.434975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.434953 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-metrics-certs\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.435163 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.435130 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.449033 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.449003 5824 generic.go:356] "Generic (PLEG): container finished" podID="f080aad9-b186-4a5e-be3a-597a046e836a" containerID="18e5972ef39a2376561ded34314c9a02b31e57d9518b2fc0aa1ef3310c571196" exitCode=0 Oct 30 14:42:19.449388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.449082 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" event={"ID":"f080aad9-b186-4a5e-be3a-597a046e836a","Type":"ContainerDied","Data":"18e5972ef39a2376561ded34314c9a02b31e57d9518b2fc0aa1ef3310c571196"} Oct 30 14:42:19.536468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.536432 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-metrics-certs\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.536874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.536742 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.536874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.536798 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkt7\" (UniqueName: \"kubernetes.io/projected/c3eebbff-a656-4bc8-a369-3da7d76830f7-kube-api-access-nbkt7\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.537153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.536913 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3eebbff-a656-4bc8-a369-3da7d76830f7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.537153 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:42:19.537019 5824 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-serving-cert: secret "maas-api-e2e-ipp-b-3b95dd-serving-cert" not found Oct 30 14:42:19.537153 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:42:19.537094 5824 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls podName:c3eebbff-a656-4bc8-a369-3da7d76830f7 nodeName:}" failed. No retries permitted until 2026-10-30 14:42:20.037070021 +0000 UTC m=+3169.837575465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls") pod "maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" (UID: "c3eebbff-a656-4bc8-a369-3da7d76830f7") : secret "maas-api-e2e-ipp-b-3b95dd-serving-cert" not found Oct 30 14:42:19.537878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.537852 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3eebbff-a656-4bc8-a369-3da7d76830f7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.539804 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.539782 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-metrics-certs\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:19.565675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:19.565642 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkt7\" (UniqueName: \"kubernetes.io/projected/c3eebbff-a656-4bc8-a369-3da7d76830f7-kube-api-access-nbkt7\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:20.045829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.045790 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:20.048309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.048287 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:20.241882 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.241843 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:20.545104 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.545045 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" podUID="87c4642c-5571-4239-aab5-0c5a254102a3" containerName="payload-pre-processing" containerID="cri-o://910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f" gracePeriod=25 Oct 30 14:42:20.576924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.576891 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z"] Oct 30 14:42:20.601248 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:42:20.601213 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3eebbff_a656_4bc8_a369_3da7d76830f7.slice/crio-c479d3f2d6311e95c4da8037abcfcde306c6e785f95c50716d30ab54a0bd05a9 WatchSource:0}: Error finding container c479d3f2d6311e95c4da8037abcfcde306c6e785f95c50716d30ab54a0bd05a9: Status 404 returned error can't find the container with id c479d3f2d6311e95c4da8037abcfcde306c6e785f95c50716d30ab54a0bd05a9 Oct 30 14:42:20.691236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.690369 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:20.789417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.787459 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-openshift-ca\") pod \"f080aad9-b186-4a5e-be3a-597a046e836a\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " Oct 30 14:42:20.789417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.787799 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f080aad9-b186-4a5e-be3a-597a046e836a-kube-api-access-ggxc5\" (UniqueName: \"kubernetes.io/projected/f080aad9-b186-4a5e-be3a-597a046e836a-kube-api-access-ggxc5\") pod \"f080aad9-b186-4a5e-be3a-597a046e836a\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " Oct 30 14:42:20.789417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.787909 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-xks-ca\") pod \"f080aad9-b186-4a5e-be3a-597a046e836a\" (UID: \"f080aad9-b186-4a5e-be3a-597a046e836a\") " Oct 30 14:42:20.789417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.789026 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-openshift-ca" pod "f080aad9-b186-4a5e-be3a-597a046e836a" (UID: "f080aad9-b186-4a5e-be3a-597a046e836a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:42:20.798205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.794683 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f080aad9-b186-4a5e-be3a-597a046e836a-kube-api-access-ggxc5" pod "f080aad9-b186-4a5e-be3a-597a046e836a" (UID: "f080aad9-b186-4a5e-be3a-597a046e836a"). InnerVolumeSpecName "kube-api-access-ggxc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:42:20.798205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.794904 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-xks-ca" pod "f080aad9-b186-4a5e-be3a-597a046e836a" (UID: "f080aad9-b186-4a5e-be3a-597a046e836a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:42:20.891181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.891151 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxc5\" (UniqueName: \"kubernetes.io/projected/f080aad9-b186-4a5e-be3a-597a046e836a-kube-api-access-ggxc5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:20.891181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.891180 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:20.891349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.891207 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f080aad9-b186-4a5e-be3a-597a046e836a-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:20.895250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.895227 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:42:20.992118 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.992085 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c4642c-5571-4239-aab5-0c5a254102a3-kube-api-access-74g49\" (UniqueName: \"kubernetes.io/projected/87c4642c-5571-4239-aab5-0c5a254102a3-kube-api-access-74g49\") pod \"87c4642c-5571-4239-aab5-0c5a254102a3\" (UID: \"87c4642c-5571-4239-aab5-0c5a254102a3\") " Oct 30 14:42:20.992383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.992249 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c4642c-5571-4239-aab5-0c5a254102a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c4642c-5571-4239-aab5-0c5a254102a3-plugins-config-volume\") pod \"87c4642c-5571-4239-aab5-0c5a254102a3\" (UID: \"87c4642c-5571-4239-aab5-0c5a254102a3\") " Oct 30 14:42:20.992851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.992825 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c4642c-5571-4239-aab5-0c5a254102a3-plugins-config-volume" pod "87c4642c-5571-4239-aab5-0c5a254102a3" (UID: "87c4642c-5571-4239-aab5-0c5a254102a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:42:20.994358 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:20.994334 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c4642c-5571-4239-aab5-0c5a254102a3-kube-api-access-74g49" pod "87c4642c-5571-4239-aab5-0c5a254102a3" (UID: "87c4642c-5571-4239-aab5-0c5a254102a3"). InnerVolumeSpecName "kube-api-access-74g49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:42:21.094817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.094775 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74g49\" (UniqueName: \"kubernetes.io/projected/87c4642c-5571-4239-aab5-0c5a254102a3-kube-api-access-74g49\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:21.094817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.094809 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c4642c-5571-4239-aab5-0c5a254102a3-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:42:21.498333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.498160 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" event={"ID":"c3eebbff-a656-4bc8-a369-3da7d76830f7","Type":"ContainerStarted","Data":"e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc"} Oct 30 14:42:21.498333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.498246 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" event={"ID":"c3eebbff-a656-4bc8-a369-3da7d76830f7","Type":"ContainerStarted","Data":"c479d3f2d6311e95c4da8037abcfcde306c6e785f95c50716d30ab54a0bd05a9"} Oct 30 14:42:21.498333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.498292 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:21.505781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.505748 5824 generic.go:356] "Generic (PLEG): container finished" podID="87c4642c-5571-4239-aab5-0c5a254102a3" containerID="910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f" exitCode=0 Oct 30 14:42:21.506075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.506051 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" event={"ID":"87c4642c-5571-4239-aab5-0c5a254102a3","Type":"ContainerDied","Data":"910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f"} Oct 30 14:42:21.506262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.506237 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" event={"ID":"87c4642c-5571-4239-aab5-0c5a254102a3","Type":"ContainerDied","Data":"f0b68acb1d72543a913a45fd5cadf006ec6b46b318643522ab41731d80647ca6"} Oct 30 14:42:21.506448 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.506434 5824 scope.go:122] "RemoveContainer" containerID="910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f" Oct 30 14:42:21.511676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.506696 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv" Oct 30 14:42:21.587831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.587769 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" podStartSLOduration=2.178545491 podStartE2EDuration="2.587749287s" totalImagesPullingTime="409.203796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:42:21.556830069 +0000 UTC m=+3171.357335550" watchObservedRunningTime="2026-10-30 14:42:21.587749287 +0000 UTC m=+3171.388254733" Oct 30 14:42:21.595077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.589507 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" event={"ID":"f080aad9-b186-4a5e-be3a-597a046e836a","Type":"ContainerDied","Data":"a1f02851c2ca455b1f80e63f4bfbc4a756437d58087d2762aba649f10245949f"} Oct 30 14:42:21.595344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.595322 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f02851c2ca455b1f80e63f4bfbc4a756437d58087d2762aba649f10245949f" Oct 30 14:42:21.595483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.589583 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d" Oct 30 14:42:21.658814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.658501 5824 scope.go:122] "RemoveContainer" containerID="910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f" Oct 30 14:42:21.659155 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:42:21.659054 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f\": container with ID starting with 910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f not found: ID does not exist" containerID="910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f" Oct 30 14:42:21.659155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.659089 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f"} err="failed to get container status \"910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f\": rpc error: code = NotFound desc = could not find container \"910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f\": container with ID starting with 910415ef5e9abad2da1a4d129bf687c41e19a502e63ab2f51e00425695dc166f not found: ID does not exist" Oct 30 14:42:21.691285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.691258 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv"] Oct 30 14:42:21.695661 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:21.695637 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6176c3-f464c4b59-qkgkv"] Oct 30 14:42:22.320140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:22.320105 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c4642c-5571-4239-aab5-0c5a254102a3" path="/var/lib/kubelet/pods/87c4642c-5571-4239-aab5-0c5a254102a3/volumes" Oct 30 14:42:27.606735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:27.606704 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:42:33.642034 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:42:33.642000 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:42:37.528801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.528767 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk"] Oct 30 14:42:37.529601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.529584 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f080aad9-b186-4a5e-be3a-597a046e836a" containerName="revoke-keys" Oct 30 14:42:37.529654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.529602 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f080aad9-b186-4a5e-be3a-597a046e836a" containerName="revoke-keys" Oct 30 14:42:37.529654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.529622 5824 cpu_manager.go:414] "containerMap: removing container" podUID="87c4642c-5571-4239-aab5-0c5a254102a3" containerName="payload-pre-processing" Oct 30 14:42:37.529654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.529627 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c4642c-5571-4239-aab5-0c5a254102a3" containerName="payload-pre-processing" Oct 30 14:42:37.529767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.529758 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f080aad9-b186-4a5e-be3a-597a046e836a" containerName="revoke-keys" Oct 30 14:42:37.529808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.529771 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c4642c-5571-4239-aab5-0c5a254102a3" containerName="payload-pre-processing" Oct 30 14:42:37.533509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.533488 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:37.555751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.555722 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk"] Oct 30 14:42:37.563827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.563799 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d05e21\"" Oct 30 14:42:37.639911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.639867 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk\" (UID: \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:37.640325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.640285 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkb4\" (UniqueName: \"kubernetes.io/projected/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-kube-api-access-stkb4\") pod \"payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk\" (UID: \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:37.764253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.744397 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk\" (UID: \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:37.765038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.765004 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkb4\" (UniqueName: \"kubernetes.io/projected/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-kube-api-access-stkb4\") pod \"payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk\" (UID: \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:37.765654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.748940 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk\" (UID: \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:37.813921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.813844 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkb4\" (UniqueName: \"kubernetes.io/projected/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-kube-api-access-stkb4\") pod \"payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk\" (UID: \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:37.848997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:37.848959 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:38.167170 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:42:38.167127 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77393ac8_1ac3_4d4a_bbe8_7aeac3b0580e.slice/crio-c0b793a0a27445e96d3ac08dc1e54d17560a1c6b735b886f733e523efedc4987 WatchSource:0}: Error finding container c0b793a0a27445e96d3ac08dc1e54d17560a1c6b735b886f733e523efedc4987: Status 404 returned error can't find the container with id c0b793a0a27445e96d3ac08dc1e54d17560a1c6b735b886f733e523efedc4987 Oct 30 14:42:38.168391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:38.168367 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk"] Oct 30 14:42:38.770802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:38.770757 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" event={"ID":"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e","Type":"ContainerStarted","Data":"8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004"} Oct 30 14:42:38.771308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:38.770819 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:38.771308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:38.770842 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" event={"ID":"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e","Type":"ContainerStarted","Data":"c0b793a0a27445e96d3ac08dc1e54d17560a1c6b735b886f733e523efedc4987"} Oct 30 14:42:38.819409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:38.819357 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" podStartSLOduration=1.819342263 podStartE2EDuration="1.819342263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:42:38.815406091 +0000 UTC m=+3188.615911556" watchObservedRunningTime="2026-10-30 14:42:38.819342263 +0000 UTC m=+3188.619847707" Oct 30 14:42:49.785106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:49.785077 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:42:53.191273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:53.191230 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg"] Oct 30 14:42:53.204481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:53.204445 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d3a829-mdjpg"] Oct 30 14:42:54.319123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:42:54.319082 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06fd4a14-1587-4b2a-b925-082a144415c2" path="/var/lib/kubelet/pods/06fd4a14-1587-4b2a-b925-082a144415c2/volumes" Oct 30 14:43:03.640973 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:43:03.640937 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:43:33.646131 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:43:33.646075 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:43:48.999109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:43:48.999033 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd"] Oct 30 14:43:49.034068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:43:49.034029 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk"] Oct 30 14:43:50.521418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:43:50.521375 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6"] Oct 30 14:43:50.556466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:43:50.556434 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn"] Oct 30 14:43:53.999679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:43:53.999637 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" podUID="8b77a2a8-2324-44f6-a28e-97170eca645c" containerName="payload-processing" containerID="cri-o://ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4" gracePeriod=25 Oct 30 14:43:54.035211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:43:54.035153 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" podUID="e4d62c9a-1dae-4303-8e00-cf0ce0a20268" containerName="payload-pre-processing" containerID="cri-o://2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027" gracePeriod=25 Oct 30 14:43:55.522685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:43:55.522632 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" podUID="b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" containerName="payload-processing" containerID="cri-o://4ed4391997996e7bf0c7628e87ce1182a838fc42d81ad59b74706cfc6bf7aaac" gracePeriod=25 Oct 30 14:43:55.557671 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:43:55.557637 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" podUID="e17d5a07-9932-42b2-b544-43bb97373bae" containerName="payload-pre-processing" containerID="cri-o://a2821e2da519a57979fd0aee314ffba06171c4930202a6d7e78a281b3d3b0179" gracePeriod=25 Oct 30 14:44:01.613105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:01.613064 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" podUID="8b77a2a8-2324-44f6-a28e-97170eca645c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:44:01.613631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:01.613110 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" podUID="e4d62c9a-1dae-4303-8e00-cf0ce0a20268" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:44:03.645317 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:44:03.645284 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:44:05.482925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:05.482887 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" podUID="e17d5a07-9932-42b2-b544-43bb97373bae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:44:05.482925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:05.482910 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" podUID="b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:44:11.613399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:11.613356 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" podUID="e4d62c9a-1dae-4303-8e00-cf0ce0a20268" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:44:11.613864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:11.613365 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" podUID="8b77a2a8-2324-44f6-a28e-97170eca645c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:44:14.321526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.321497 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:44:14.385000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.384971 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:44:14.465747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.465707 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-plugins-config-volume\") pod \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\" (UID: \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\") " Oct 30 14:44:14.465942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.465775 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b77a2a8-2324-44f6-a28e-97170eca645c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b77a2a8-2324-44f6-a28e-97170eca645c-plugins-config-volume\") pod \"8b77a2a8-2324-44f6-a28e-97170eca645c\" (UID: \"8b77a2a8-2324-44f6-a28e-97170eca645c\") " Oct 30 14:44:14.466018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.465951 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b77a2a8-2324-44f6-a28e-97170eca645c-kube-api-access-scdwj\" (UniqueName: \"kubernetes.io/projected/8b77a2a8-2324-44f6-a28e-97170eca645c-kube-api-access-scdwj\") pod \"8b77a2a8-2324-44f6-a28e-97170eca645c\" (UID: \"8b77a2a8-2324-44f6-a28e-97170eca645c\") " Oct 30 14:44:14.466088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.466021 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-kube-api-access-4twfp\" (UniqueName: \"kubernetes.io/projected/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-kube-api-access-4twfp\") pod \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\" (UID: \"e4d62c9a-1dae-4303-8e00-cf0ce0a20268\") " Oct 30 14:44:14.466258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.466227 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-plugins-config-volume" pod "e4d62c9a-1dae-4303-8e00-cf0ce0a20268" (UID: "e4d62c9a-1dae-4303-8e00-cf0ce0a20268"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:44:14.466428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.466403 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b77a2a8-2324-44f6-a28e-97170eca645c-plugins-config-volume" pod "8b77a2a8-2324-44f6-a28e-97170eca645c" (UID: "8b77a2a8-2324-44f6-a28e-97170eca645c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:44:14.467229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.467206 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:14.467327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.467231 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b77a2a8-2324-44f6-a28e-97170eca645c-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:14.468870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.468840 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b77a2a8-2324-44f6-a28e-97170eca645c-kube-api-access-scdwj" pod "8b77a2a8-2324-44f6-a28e-97170eca645c" (UID: "8b77a2a8-2324-44f6-a28e-97170eca645c"). InnerVolumeSpecName "kube-api-access-scdwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:44:14.468961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.468886 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-kube-api-access-4twfp" pod "e4d62c9a-1dae-4303-8e00-cf0ce0a20268" (UID: "e4d62c9a-1dae-4303-8e00-cf0ce0a20268"). InnerVolumeSpecName "kube-api-access-4twfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:44:14.568863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.568772 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scdwj\" (UniqueName: \"kubernetes.io/projected/8b77a2a8-2324-44f6-a28e-97170eca645c-kube-api-access-scdwj\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:14.568863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.568807 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4twfp\" (UniqueName: \"kubernetes.io/projected/e4d62c9a-1dae-4303-8e00-cf0ce0a20268-kube-api-access-4twfp\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:14.664169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.664132 5824 generic.go:356] "Generic (PLEG): container finished" podID="8b77a2a8-2324-44f6-a28e-97170eca645c" containerID="ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4" exitCode=0 Oct 30 14:44:14.664470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.664215 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" Oct 30 14:44:14.664470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.664243 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" event={"ID":"8b77a2a8-2324-44f6-a28e-97170eca645c","Type":"ContainerDied","Data":"ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4"} Oct 30 14:44:14.664470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.664365 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd" event={"ID":"8b77a2a8-2324-44f6-a28e-97170eca645c","Type":"ContainerDied","Data":"15b865fc9846eea860c13d198d1b72abd2859f5d3351b5c012b40eb6ecb8e91e"} Oct 30 14:44:14.664470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.664427 5824 scope.go:122] "RemoveContainer" containerID="ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4" Oct 30 14:44:14.666155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.666131 5824 generic.go:356] "Generic (PLEG): container finished" podID="e4d62c9a-1dae-4303-8e00-cf0ce0a20268" containerID="2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027" exitCode=0 Oct 30 14:44:14.666412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.666164 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" event={"ID":"e4d62c9a-1dae-4303-8e00-cf0ce0a20268","Type":"ContainerDied","Data":"2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027"} Oct 30 14:44:14.666412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.666250 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" event={"ID":"e4d62c9a-1dae-4303-8e00-cf0ce0a20268","Type":"ContainerDied","Data":"97c170f9a85c4d3b96c4e797bd1d0a7ee7c2ae46e9c9a089f43d89cfd956006e"} Oct 30 14:44:14.666630 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.666608 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk" Oct 30 14:44:14.681655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.681322 5824 scope.go:122] "RemoveContainer" containerID="ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4" Oct 30 14:44:14.682269 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:44:14.682249 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4\": container with ID starting with ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4 not found: ID does not exist" containerID="ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4" Oct 30 14:44:14.682386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.682276 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4"} err="failed to get container status \"ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4\": rpc error: code = NotFound desc = could not find container \"ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4\": container with ID starting with ee7cfc9caa25dbd7395d91c1509cef83f2f24e389d3aabbac4d885c5ede7d4d4 not found: ID does not exist" Oct 30 14:44:14.682386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.682295 5824 scope.go:122] "RemoveContainer" containerID="2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027" Oct 30 14:44:14.692993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.692974 5824 scope.go:122] "RemoveContainer" containerID="2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027" Oct 30 14:44:14.693267 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:44:14.693251 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027\": container with ID starting with 2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027 not found: ID does not exist" containerID="2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027" Oct 30 14:44:14.693319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.693275 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027"} err="failed to get container status \"2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027\": rpc error: code = NotFound desc = could not find container \"2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027\": container with ID starting with 2229bf0ed0cd2091b980c915b97d08397cd0bceff9c6beffee8f270a83d4b027 not found: ID does not exist" Oct 30 14:44:14.728075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.728046 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd"] Oct 30 14:44:14.741044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.741018 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-acefbb-569588ff76-rxwvd"] Oct 30 14:44:14.772067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.772043 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk"] Oct 30 14:44:14.780498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:14.780476 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-acefbb-5c87485f47-l4jfk"] Oct 30 14:44:15.482472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.482432 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" podUID="b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:44:15.482851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.482453 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" podUID="e17d5a07-9932-42b2-b544-43bb97373bae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:44:15.678248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.678181 5824 generic.go:356] "Generic (PLEG): container finished" podID="e17d5a07-9932-42b2-b544-43bb97373bae" containerID="a2821e2da519a57979fd0aee314ffba06171c4930202a6d7e78a281b3d3b0179" exitCode=0 Oct 30 14:44:15.678368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.678239 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" event={"ID":"e17d5a07-9932-42b2-b544-43bb97373bae","Type":"ContainerDied","Data":"a2821e2da519a57979fd0aee314ffba06171c4930202a6d7e78a281b3d3b0179"} Oct 30 14:44:15.680230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.680201 5824 generic.go:356] "Generic (PLEG): container finished" podID="b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" containerID="4ed4391997996e7bf0c7628e87ce1182a838fc42d81ad59b74706cfc6bf7aaac" exitCode=0 Oct 30 14:44:15.680331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.680260 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" event={"ID":"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d","Type":"ContainerDied","Data":"4ed4391997996e7bf0c7628e87ce1182a838fc42d81ad59b74706cfc6bf7aaac"} Oct 30 14:44:15.854001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.853875 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:44:15.861137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.861116 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:44:15.992834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.992795 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17d5a07-9932-42b2-b544-43bb97373bae-kube-api-access-twfh9\" (UniqueName: \"kubernetes.io/projected/e17d5a07-9932-42b2-b544-43bb97373bae-kube-api-access-twfh9\") pod \"e17d5a07-9932-42b2-b544-43bb97373bae\" (UID: \"e17d5a07-9932-42b2-b544-43bb97373bae\") " Oct 30 14:44:15.993032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.992942 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-kube-api-access-kz7qj\" (UniqueName: \"kubernetes.io/projected/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-kube-api-access-kz7qj\") pod \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\" (UID: \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\") " Oct 30 14:44:15.993032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.992970 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-plugins-config-volume\") pod \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\" (UID: \"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d\") " Oct 30 14:44:15.993032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.992990 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17d5a07-9932-42b2-b544-43bb97373bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17d5a07-9932-42b2-b544-43bb97373bae-plugins-config-volume\") pod \"e17d5a07-9932-42b2-b544-43bb97373bae\" (UID: \"e17d5a07-9932-42b2-b544-43bb97373bae\") " Oct 30 14:44:15.993463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.993430 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-plugins-config-volume" pod "b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" (UID: "b833a15f-4b7c-4c73-93c5-cd133c8c2b2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:44:15.993657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.993636 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:15.993845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.993825 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17d5a07-9932-42b2-b544-43bb97373bae-plugins-config-volume" pod "e17d5a07-9932-42b2-b544-43bb97373bae" (UID: "e17d5a07-9932-42b2-b544-43bb97373bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:44:15.995480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.995459 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17d5a07-9932-42b2-b544-43bb97373bae-kube-api-access-twfh9" pod "e17d5a07-9932-42b2-b544-43bb97373bae" (UID: "e17d5a07-9932-42b2-b544-43bb97373bae"). InnerVolumeSpecName "kube-api-access-twfh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:44:15.995573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:15.995514 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-kube-api-access-kz7qj" pod "b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" (UID: "b833a15f-4b7c-4c73-93c5-cd133c8c2b2d"). InnerVolumeSpecName "kube-api-access-kz7qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:44:16.095068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.095032 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz7qj\" (UniqueName: \"kubernetes.io/projected/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d-kube-api-access-kz7qj\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:16.095068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.095062 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e17d5a07-9932-42b2-b544-43bb97373bae-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:16.095068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.095071 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twfh9\" (UniqueName: \"kubernetes.io/projected/e17d5a07-9932-42b2-b544-43bb97373bae-kube-api-access-twfh9\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:16.319143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.319052 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b77a2a8-2324-44f6-a28e-97170eca645c" path="/var/lib/kubelet/pods/8b77a2a8-2324-44f6-a28e-97170eca645c/volumes" Oct 30 14:44:16.319558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.319543 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d62c9a-1dae-4303-8e00-cf0ce0a20268" path="/var/lib/kubelet/pods/e4d62c9a-1dae-4303-8e00-cf0ce0a20268/volumes" Oct 30 14:44:16.691270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.691238 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" Oct 30 14:44:16.691270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.691248 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6" event={"ID":"b833a15f-4b7c-4c73-93c5-cd133c8c2b2d","Type":"ContainerDied","Data":"9bd6776820aa8f3c26cb7b7625dbc193c8d955a320084d15202019ed0b542c96"} Oct 30 14:44:16.691818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.691307 5824 scope.go:122] "RemoveContainer" containerID="4ed4391997996e7bf0c7628e87ce1182a838fc42d81ad59b74706cfc6bf7aaac" Oct 30 14:44:16.692981 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.692955 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" event={"ID":"e17d5a07-9932-42b2-b544-43bb97373bae","Type":"ContainerDied","Data":"918e9868fc62b42f010ced270394bd7c80b2f8dd58054947d77fe9438e3f78ae"} Oct 30 14:44:16.693096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.692959 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn" Oct 30 14:44:16.703334 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.703314 5824 scope.go:122] "RemoveContainer" containerID="a2821e2da519a57979fd0aee314ffba06171c4930202a6d7e78a281b3d3b0179" Oct 30 14:44:16.754207 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.754170 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6"] Oct 30 14:44:16.767642 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.767619 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e50e63-585ff6bcbd-jlsv6"] Oct 30 14:44:16.809136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.809104 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn"] Oct 30 14:44:16.812373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:16.812351 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e50e63-7674c8c9f5-nggnn"] Oct 30 14:44:18.320492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:18.320450 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" path="/var/lib/kubelet/pods/b833a15f-4b7c-4c73-93c5-cd133c8c2b2d/volumes" Oct 30 14:44:18.321071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:18.321049 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17d5a07-9932-42b2-b544-43bb97373bae" path="/var/lib/kubelet/pods/e17d5a07-9932-42b2-b544-43bb97373bae/volumes" Oct 30 14:44:21.564563 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.564530 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j"] Oct 30 14:44:21.565507 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565487 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" containerName="payload-processing" Oct 30 14:44:21.565507 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565507 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" containerName="payload-processing" Oct 30 14:44:21.565701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565534 5824 cpu_manager.go:414] "containerMap: removing container" podUID="8b77a2a8-2324-44f6-a28e-97170eca645c" containerName="payload-processing" Oct 30 14:44:21.565701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565542 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b77a2a8-2324-44f6-a28e-97170eca645c" containerName="payload-processing" Oct 30 14:44:21.565701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565574 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e4d62c9a-1dae-4303-8e00-cf0ce0a20268" containerName="payload-pre-processing" Oct 30 14:44:21.565701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565583 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d62c9a-1dae-4303-8e00-cf0ce0a20268" containerName="payload-pre-processing" Oct 30 14:44:21.565701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565632 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e17d5a07-9932-42b2-b544-43bb97373bae" containerName="payload-pre-processing" Oct 30 14:44:21.565701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565639 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17d5a07-9932-42b2-b544-43bb97373bae" containerName="payload-pre-processing" Oct 30 14:44:21.566026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565796 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b77a2a8-2324-44f6-a28e-97170eca645c" containerName="payload-processing" Oct 30 14:44:21.566026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565816 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d62c9a-1dae-4303-8e00-cf0ce0a20268" containerName="payload-pre-processing" Oct 30 14:44:21.566026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565839 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17d5a07-9932-42b2-b544-43bb97373bae" containerName="payload-pre-processing" Oct 30 14:44:21.566026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.565848 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b833a15f-4b7c-4c73-93c5-cd133c8c2b2d" containerName="payload-processing" Oct 30 14:44:21.583575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.583532 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.594524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.594490 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j"] Oct 30 14:44:21.603235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.602180 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d05e21-openshift-default-dockercfg-6p687\"" Oct 30 14:44:21.684837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.684801 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f073fd1c-086a-4336-80e4-c3699dd224ad-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.685015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.684908 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.685015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.684950 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.685103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.685021 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.685103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.685091 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.685222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.685140 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.685222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.685165 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.685222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.685199 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vq7r\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-kube-api-access-9vq7r\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.685222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.685217 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.789159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.789125 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.789945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.789919 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.790135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.790118 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.790325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.790306 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.790482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.790450 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vq7r\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-kube-api-access-9vq7r\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.790616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.790600 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.790896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.790880 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f073fd1c-086a-4336-80e4-c3699dd224ad-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.791085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.791069 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.791236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.791220 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.791794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.791774 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.792155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.792136 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.792569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.792549 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.818013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.817631 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f073fd1c-086a-4336-80e4-c3699dd224ad-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.818013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.817928 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.840246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.837487 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.840246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.838021 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.877213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.869036 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vq7r\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-kube-api-access-9vq7r\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.877213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.869631 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:21.902769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:21.902733 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:22.238379 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:44:22.238350 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf073fd1c_086a_4336_80e4_c3699dd224ad.slice/crio-b887060e73b6e0e562f83872dcb6da5faba5e53df106ab37aa4d3e3693a76cf0 WatchSource:0}: Error finding container b887060e73b6e0e562f83872dcb6da5faba5e53df106ab37aa4d3e3693a76cf0: Status 404 returned error can't find the container with id b887060e73b6e0e562f83872dcb6da5faba5e53df106ab37aa4d3e3693a76cf0 Oct 30 14:44:22.239350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:22.239325 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j"] Oct 30 14:44:22.241430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:22.241394 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:44:22.241508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:22.241465 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:44:22.241508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:22.241495 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:44:22.806392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:22.806347 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" event={"ID":"f073fd1c-086a-4336-80e4-c3699dd224ad","Type":"ContainerStarted","Data":"0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d"} Oct 30 14:44:22.806787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:22.806420 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" event={"ID":"f073fd1c-086a-4336-80e4-c3699dd224ad","Type":"ContainerStarted","Data":"b887060e73b6e0e562f83872dcb6da5faba5e53df106ab37aa4d3e3693a76cf0"} Oct 30 14:44:22.897161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:22.897108 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" podStartSLOduration=1.897095889 podStartE2EDuration="1.897095889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:44:22.880650999 +0000 UTC m=+3292.681156465" watchObservedRunningTime="2026-10-30 14:44:22.897095889 +0000 UTC m=+3292.697601332" Oct 30 14:44:22.902835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:22.902807 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:23.905128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:23.905090 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 14:44:23.905537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:23.905150 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 14:44:24.904047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:24.904008 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 14:44:24.904267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:24.904078 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 14:44:25.903298 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:25.903261 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 14:44:25.903694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:25.903324 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 14:44:26.903987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:26.903955 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 14:44:26.904415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:26.904007 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 14:44:27.903320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:27.903288 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 14:44:27.903516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:27.903338 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 14:44:28.908630 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:28.908599 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:28.909047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:28.908895 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:28.909708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:28.909689 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:44:33.641849 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:44:33.641816 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:44:48.124808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.124775 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm"] Oct 30 14:44:48.125468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.125023 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" podUID="fb511c01-90a9-465a-8b03-830417169b98" containerName="maas-api" containerID="cri-o://517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff" gracePeriod=30 Oct 30 14:44:48.465532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.465501 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:44:48.552531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.552496 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb511c01-90a9-465a-8b03-830417169b98-kube-api-access-8m4mv\" (UniqueName: \"kubernetes.io/projected/fb511c01-90a9-465a-8b03-830417169b98-kube-api-access-8m4mv\") pod \"fb511c01-90a9-465a-8b03-830417169b98\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " Oct 30 14:44:48.552531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.552537 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb511c01-90a9-465a-8b03-830417169b98-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb511c01-90a9-465a-8b03-830417169b98-openshift-service-ca\") pod \"fb511c01-90a9-465a-8b03-830417169b98\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " Oct 30 14:44:48.552803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.552594 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-metrics-certs\") pod \"fb511c01-90a9-465a-8b03-830417169b98\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " Oct 30 14:44:48.552803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.552648 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-maas-api-tls\") pod \"fb511c01-90a9-465a-8b03-830417169b98\" (UID: \"fb511c01-90a9-465a-8b03-830417169b98\") " Oct 30 14:44:48.570782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.570300 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb511c01-90a9-465a-8b03-830417169b98-openshift-service-ca" pod "fb511c01-90a9-465a-8b03-830417169b98" (UID: "fb511c01-90a9-465a-8b03-830417169b98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:44:48.574363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.574311 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-maas-api-tls" pod "fb511c01-90a9-465a-8b03-830417169b98" (UID: "fb511c01-90a9-465a-8b03-830417169b98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:44:48.574520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.574406 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-metrics-certs" pod "fb511c01-90a9-465a-8b03-830417169b98" (UID: "fb511c01-90a9-465a-8b03-830417169b98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:44:48.576450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.576423 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb511c01-90a9-465a-8b03-830417169b98-kube-api-access-8m4mv" pod "fb511c01-90a9-465a-8b03-830417169b98" (UID: "fb511c01-90a9-465a-8b03-830417169b98"). InnerVolumeSpecName "kube-api-access-8m4mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:44:48.655663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.655572 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m4mv\" (UniqueName: \"kubernetes.io/projected/fb511c01-90a9-465a-8b03-830417169b98-kube-api-access-8m4mv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:48.655663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.655618 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb511c01-90a9-465a-8b03-830417169b98-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:48.655663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.655640 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:48.655663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.655660 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb511c01-90a9-465a-8b03-830417169b98-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:48.799089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:48.799055 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk"] Oct 30 14:44:49.042983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.042953 5824 generic.go:356] "Generic (PLEG): container finished" podID="fb511c01-90a9-465a-8b03-830417169b98" containerID="517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff" exitCode=0 Oct 30 14:44:49.043183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.043002 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" Oct 30 14:44:49.043183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.043039 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" event={"ID":"fb511c01-90a9-465a-8b03-830417169b98","Type":"ContainerDied","Data":"517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff"} Oct 30 14:44:49.043183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.043091 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" event={"ID":"fb511c01-90a9-465a-8b03-830417169b98","Type":"ContainerDied","Data":"6aee3028708dcff412763d6f9320bff9955ce0a929bed9efa7be7e6d567f2894"} Oct 30 14:44:49.043183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.043114 5824 scope.go:122] "RemoveContainer" containerID="517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff" Oct 30 14:44:49.054520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.054503 5824 scope.go:122] "RemoveContainer" containerID="517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff" Oct 30 14:44:49.054800 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:44:49.054784 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff\": container with ID starting with 517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff not found: ID does not exist" containerID="517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff" Oct 30 14:44:49.054839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.054819 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff"} err="failed to get container status \"517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff\": rpc error: code = NotFound desc = could not find container \"517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff\": container with ID starting with 517ce1e814fb07fbdc715b65f9bae42d39cc2a307ce3f86f9cd36d63d81df8ff not found: ID does not exist" Oct 30 14:44:49.107053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.107026 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm"] Oct 30 14:44:49.110557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:49.110533 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm"] Oct 30 14:44:50.319336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:50.319302 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb511c01-90a9-465a-8b03-830417169b98" path="/var/lib/kubelet/pods/fb511c01-90a9-465a-8b03-830417169b98/volumes" Oct 30 14:44:51.423920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:51.423875 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-acefbb-74d6cd74bc-rc6jm" podUID="fb511c01-90a9-465a-8b03-830417169b98" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.166:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:44:53.800338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:53.800298 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" podUID="77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e" containerName="payload-processing" containerID="cri-o://8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004" gracePeriod=25 Oct 30 14:44:54.070629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.070564 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:44:54.080963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.080936 5824 generic.go:356] "Generic (PLEG): container finished" podID="77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e" containerID="8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004" exitCode=0 Oct 30 14:44:54.081102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.080976 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" Oct 30 14:44:54.081102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.081022 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" event={"ID":"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e","Type":"ContainerDied","Data":"8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004"} Oct 30 14:44:54.081102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.081081 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk" event={"ID":"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e","Type":"ContainerDied","Data":"c0b793a0a27445e96d3ac08dc1e54d17560a1c6b735b886f733e523efedc4987"} Oct 30 14:44:54.081242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.081108 5824 scope.go:122] "RemoveContainer" containerID="8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004" Oct 30 14:44:54.091661 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.091641 5824 scope.go:122] "RemoveContainer" containerID="8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004" Oct 30 14:44:54.091927 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:44:54.091906 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004\": container with ID starting with 8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004 not found: ID does not exist" containerID="8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004" Oct 30 14:44:54.091996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.091937 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004"} err="failed to get container status \"8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004\": rpc error: code = NotFound desc = could not find container \"8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004\": container with ID starting with 8966f67be076bf3a59f4776df94bf8297b83c37830021c842471197e22a5c004 not found: ID does not exist" Oct 30 14:44:54.163887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.163856 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-plugins-config-volume\") pod \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\" (UID: \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\") " Oct 30 14:44:54.164058 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.163993 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-kube-api-access-stkb4\" (UniqueName: \"kubernetes.io/projected/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-kube-api-access-stkb4\") pod \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\" (UID: \"77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e\") " Oct 30 14:44:54.164389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.164359 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-plugins-config-volume" pod "77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e" (UID: "77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:44:54.164828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.164804 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:54.166225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.166164 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-kube-api-access-stkb4" pod "77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e" (UID: "77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e"). InnerVolumeSpecName "kube-api-access-stkb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:44:54.266063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.266028 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stkb4\" (UniqueName: \"kubernetes.io/projected/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e-kube-api-access-stkb4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:54.426689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.426657 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk"] Oct 30 14:44:54.433561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:54.433536 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d05e21-6796c5fc4d-xzzzk"] Oct 30 14:44:55.461649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.461611 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst"] Oct 30 14:44:55.462474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.462454 5824 cpu_manager.go:414] "containerMap: removing container" podUID="fb511c01-90a9-465a-8b03-830417169b98" containerName="maas-api" Oct 30 14:44:55.462474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.462474 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb511c01-90a9-465a-8b03-830417169b98" containerName="maas-api" Oct 30 14:44:55.462622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.462489 5824 cpu_manager.go:414] "containerMap: removing container" podUID="77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e" containerName="payload-processing" Oct 30 14:44:55.462622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.462494 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e" containerName="payload-processing" Oct 30 14:44:55.462714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.462623 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e" containerName="payload-processing" Oct 30 14:44:55.462714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.462644 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb511c01-90a9-465a-8b03-830417169b98" containerName="maas-api" Oct 30 14:44:55.467402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.467382 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.485169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.485139 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst"] Oct 30 14:44:55.558312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.558282 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:44:55.558312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.558310 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:44:55.588012 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.587983 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb865\" (UniqueName: \"kubernetes.io/projected/7ba98114-4665-46bf-a947-5a86f9897a3a-kube-api-access-rb865\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.588165 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.588046 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.588280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.588256 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.690129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.690096 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb865\" (UniqueName: \"kubernetes.io/projected/7ba98114-4665-46bf-a947-5a86f9897a3a-kube-api-access-rb865\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.690338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.690159 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.690338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.690286 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.690997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.690975 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.692518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.692502 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.715992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.715940 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb865\" (UniqueName: \"kubernetes.io/projected/7ba98114-4665-46bf-a947-5a86f9897a3a-kube-api-access-rb865\") pod \"maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:55.790843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:55.790814 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:56.102154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:56.102121 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst"] Oct 30 14:44:56.102438 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:44:56.102408 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba98114_4665_46bf_a947_5a86f9897a3a.slice/crio-33bc88e92828886738ea0ba12d3de185824b0ce75798036bf2e0ed64fe45cd1c WatchSource:0}: Error finding container 33bc88e92828886738ea0ba12d3de185824b0ce75798036bf2e0ed64fe45cd1c: Status 404 returned error can't find the container with id 33bc88e92828886738ea0ba12d3de185824b0ce75798036bf2e0ed64fe45cd1c Oct 30 14:44:56.319233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:56.319202 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e" path="/var/lib/kubelet/pods/77393ac8-1ac3-4d4a-bbe8-7aeac3b0580e/volumes" Oct 30 14:44:57.107873 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:57.107841 5824 generic.go:356] "Generic (PLEG): container finished" podID="7ba98114-4665-46bf-a947-5a86f9897a3a" containerID="a5a47bfb9dcf3c5581ce6e41fe0c322f9a4f0323eca8a19c0d07438de0ea2a32" exitCode=0 Oct 30 14:44:57.108352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:57.107971 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" event={"ID":"7ba98114-4665-46bf-a947-5a86f9897a3a","Type":"ContainerDied","Data":"a5a47bfb9dcf3c5581ce6e41fe0c322f9a4f0323eca8a19c0d07438de0ea2a32"} Oct 30 14:44:57.108352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:57.108054 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" event={"ID":"7ba98114-4665-46bf-a947-5a86f9897a3a","Type":"ContainerStarted","Data":"33bc88e92828886738ea0ba12d3de185824b0ce75798036bf2e0ed64fe45cd1c"} Oct 30 14:44:58.274741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.274711 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:58.341015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.340985 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-openshift-ca\") pod \"7ba98114-4665-46bf-a947-5a86f9897a3a\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " Oct 30 14:44:58.341225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.341049 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba98114-4665-46bf-a947-5a86f9897a3a-kube-api-access-rb865\" (UniqueName: \"kubernetes.io/projected/7ba98114-4665-46bf-a947-5a86f9897a3a-kube-api-access-rb865\") pod \"7ba98114-4665-46bf-a947-5a86f9897a3a\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " Oct 30 14:44:58.341225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.341096 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-xks-ca\") pod \"7ba98114-4665-46bf-a947-5a86f9897a3a\" (UID: \"7ba98114-4665-46bf-a947-5a86f9897a3a\") " Oct 30 14:44:58.341398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.341364 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-openshift-ca" pod "7ba98114-4665-46bf-a947-5a86f9897a3a" (UID: "7ba98114-4665-46bf-a947-5a86f9897a3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:44:58.341887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.341864 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:58.345346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.345316 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-xks-ca" pod "7ba98114-4665-46bf-a947-5a86f9897a3a" (UID: "7ba98114-4665-46bf-a947-5a86f9897a3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:44:58.345458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.345366 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba98114-4665-46bf-a947-5a86f9897a3a-kube-api-access-rb865" pod "7ba98114-4665-46bf-a947-5a86f9897a3a" (UID: "7ba98114-4665-46bf-a947-5a86f9897a3a"). InnerVolumeSpecName "kube-api-access-rb865". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:44:58.443727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.443696 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb865\" (UniqueName: \"kubernetes.io/projected/7ba98114-4665-46bf-a947-5a86f9897a3a-kube-api-access-rb865\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:58.443727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:58.443725 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ba98114-4665-46bf-a947-5a86f9897a3a-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:44:59.127228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:59.127198 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" Oct 30 14:44:59.127447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:59.127239 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst" event={"ID":"7ba98114-4665-46bf-a947-5a86f9897a3a","Type":"ContainerDied","Data":"33bc88e92828886738ea0ba12d3de185824b0ce75798036bf2e0ed64fe45cd1c"} Oct 30 14:44:59.127447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:44:59.127316 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33bc88e92828886738ea0ba12d3de185824b0ce75798036bf2e0ed64fe45cd1c" Oct 30 14:45:00.161137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.161036 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx"] Oct 30 14:45:00.163759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.162965 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7ba98114-4665-46bf-a947-5a86f9897a3a" containerName="revoke-keys" Oct 30 14:45:00.163759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.163005 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba98114-4665-46bf-a947-5a86f9897a3a" containerName="revoke-keys" Oct 30 14:45:00.163759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.163284 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba98114-4665-46bf-a947-5a86f9897a3a" containerName="revoke-keys" Oct 30 14:45:00.170818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.170797 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws"] Oct 30 14:45:00.170961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.170936 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:00.176801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.176776 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb"] Oct 30 14:45:00.182504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.182483 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:00.185707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.185687 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:00.188123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.188080 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf"] Oct 30 14:45:00.208299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.204987 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb"] Oct 30 14:45:00.208299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.205037 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx"] Oct 30 14:45:00.208299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.205092 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:00.214268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.209575 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws"] Oct 30 14:45:00.216198 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.216156 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:45:00.233129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.233097 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf"] Oct 30 14:45:00.246758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.246728 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh"] Oct 30 14:45:00.254929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.254901 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:00.264975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.264948 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952"] Oct 30 14:45:00.277834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.277464 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952"] Oct 30 14:45:00.277834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.277579 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.285690 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl"] Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.286757 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqmv\" (UniqueName: \"kubernetes.io/projected/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-kube-api-access-zgqmv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb\" (UID: \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.286807 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb\" (UID: \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.286839 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d3615-218b-49be-b6b6-99062913a1df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf\" (UID: \"110d3615-218b-49be-b6b6-99062913a1df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.286914 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acc1072-69bf-4a93-8428-90ba63f5dea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh\" (UID: \"7acc1072-69bf-4a93-8428-90ba63f5dea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.286970 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6770fcff-79f8-4f91-a951-308c8ed76836-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx\" (UID: \"6770fcff-79f8-4f91-a951-308c8ed76836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.287079 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws\" (UID: \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.287153 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj5pm\" (UniqueName: \"kubernetes.io/projected/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-kube-api-access-fj5pm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws\" (UID: \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.287263 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwgg\" (UniqueName: \"kubernetes.io/projected/110d3615-218b-49be-b6b6-99062913a1df-kube-api-access-mdwgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf\" (UID: \"110d3615-218b-49be-b6b6-99062913a1df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.287331 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbv47\" (UniqueName: \"kubernetes.io/projected/7acc1072-69bf-4a93-8428-90ba63f5dea8-kube-api-access-jbv47\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh\" (UID: \"7acc1072-69bf-4a93-8428-90ba63f5dea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.287367 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgd6p\" (UniqueName: \"kubernetes.io/projected/6770fcff-79f8-4f91-a951-308c8ed76836-kube-api-access-pgd6p\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx\" (UID: \"6770fcff-79f8-4f91-a951-308c8ed76836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:00.294226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.293268 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:00.301893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.301872 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh"] Oct 30 14:45:00.311417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.311385 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws"] Oct 30 14:45:00.320554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.320529 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl"] Oct 30 14:45:00.320705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.320672 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:00.352285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.345920 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w"] Oct 30 14:45:00.372724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.371799 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws"] Oct 30 14:45:00.372724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.371892 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:00.375108 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.375081 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq"] Oct 30 14:45:00.395570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.395532 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acc1072-69bf-4a93-8428-90ba63f5dea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh\" (UID: \"7acc1072-69bf-4a93-8428-90ba63f5dea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:00.395746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.395608 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5076f0fd-cf19-46e1-8cb5-ab933b424acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl\" (UID: \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:00.395746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.395650 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6770fcff-79f8-4f91-a951-308c8ed76836-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx\" (UID: \"6770fcff-79f8-4f91-a951-308c8ed76836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:00.395746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.395709 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gbrt\" (UniqueName: \"kubernetes.io/projected/5076f0fd-cf19-46e1-8cb5-ab933b424acb-kube-api-access-2gbrt\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl\" (UID: \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:00.395917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.395756 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feee7683-a6bb-47fb-b085-6a510fc1b993-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952\" (UID: \"feee7683-a6bb-47fb-b085-6a510fc1b993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:00.395917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.395811 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws\" (UID: \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:00.395917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.395880 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj5pm\" (UniqueName: \"kubernetes.io/projected/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-kube-api-access-fj5pm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws\" (UID: \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:00.396072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.395980 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwgg\" (UniqueName: \"kubernetes.io/projected/110d3615-218b-49be-b6b6-99062913a1df-kube-api-access-mdwgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf\" (UID: \"110d3615-218b-49be-b6b6-99062913a1df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:00.396072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.396060 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbv47\" (UniqueName: \"kubernetes.io/projected/7acc1072-69bf-4a93-8428-90ba63f5dea8-kube-api-access-jbv47\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh\" (UID: \"7acc1072-69bf-4a93-8428-90ba63f5dea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:00.396205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.396096 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgd6p\" (UniqueName: \"kubernetes.io/projected/6770fcff-79f8-4f91-a951-308c8ed76836-kube-api-access-pgd6p\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx\" (UID: \"6770fcff-79f8-4f91-a951-308c8ed76836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:00.396205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.396176 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5c08e1-364b-456a-9888-a26fdd1055cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws\" (UID: \"bf5c08e1-364b-456a-9888-a26fdd1055cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:00.396320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.396229 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hxb\" (UniqueName: \"kubernetes.io/projected/bf5c08e1-364b-456a-9888-a26fdd1055cb-kube-api-access-j4hxb\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws\" (UID: \"bf5c08e1-364b-456a-9888-a26fdd1055cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:00.396320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.396272 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mphs\" (UniqueName: \"kubernetes.io/projected/feee7683-a6bb-47fb-b085-6a510fc1b993-kube-api-access-6mphs\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952\" (UID: \"feee7683-a6bb-47fb-b085-6a510fc1b993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:00.396320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.396313 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqmv\" (UniqueName: \"kubernetes.io/projected/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-kube-api-access-zgqmv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb\" (UID: \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:00.396472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.396353 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb\" (UID: \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:00.396472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.396389 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d3615-218b-49be-b6b6-99062913a1df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf\" (UID: \"110d3615-218b-49be-b6b6-99062913a1df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:00.397459 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.397237 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d3615-218b-49be-b6b6-99062913a1df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf\" (UID: \"110d3615-218b-49be-b6b6-99062913a1df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:00.397459 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.397344 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:00.398914 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.398742 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl"] Oct 30 14:45:00.401835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.399293 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb\" (UID: \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:00.401835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.399362 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acc1072-69bf-4a93-8428-90ba63f5dea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh\" (UID: \"7acc1072-69bf-4a93-8428-90ba63f5dea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:00.401835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.399945 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6770fcff-79f8-4f91-a951-308c8ed76836-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx\" (UID: \"6770fcff-79f8-4f91-a951-308c8ed76836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:00.401835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.399950 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws\" (UID: \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:00.405832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.405363 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8"] Oct 30 14:45:00.421457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.421387 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:00.424101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.424017 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w"] Oct 30 14:45:00.435309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.433213 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgd6p\" (UniqueName: \"kubernetes.io/projected/6770fcff-79f8-4f91-a951-308c8ed76836-kube-api-access-pgd6p\") pod \"maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx\" (UID: \"6770fcff-79f8-4f91-a951-308c8ed76836\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:00.439065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.439022 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwgg\" (UniqueName: \"kubernetes.io/projected/110d3615-218b-49be-b6b6-99062913a1df-kube-api-access-mdwgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf\" (UID: \"110d3615-218b-49be-b6b6-99062913a1df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:00.476242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.454978 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq"] Oct 30 14:45:00.482465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.482435 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9"] Oct 30 14:45:00.484511 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.484481 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbv47\" (UniqueName: \"kubernetes.io/projected/7acc1072-69bf-4a93-8428-90ba63f5dea8-kube-api-access-jbv47\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh\" (UID: \"7acc1072-69bf-4a93-8428-90ba63f5dea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:00.508639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.508600 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj5pm\" (UniqueName: \"kubernetes.io/projected/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-kube-api-access-fj5pm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws\" (UID: \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:00.509812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.509774 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhnm5\" (UniqueName: \"kubernetes.io/projected/3069b2d7-58d3-4095-ae46-c2e8d355931a-kube-api-access-zhnm5\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8\" (UID: \"3069b2d7-58d3-4095-ae46-c2e8d355931a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:00.509947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.509874 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069b2d7-58d3-4095-ae46-c2e8d355931a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8\" (UID: \"3069b2d7-58d3-4095-ae46-c2e8d355931a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:00.509947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.509933 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4k5\" (UniqueName: \"kubernetes.io/projected/e566451a-09cf-4392-9006-23a6444553d9-kube-api-access-sf4k5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq\" (UID: \"e566451a-09cf-4392-9006-23a6444553d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:00.510050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.509963 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0041a35-bbe1-40b1-ac40-f28760c6ec17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w\" (UID: \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:00.510050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.510021 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5c08e1-364b-456a-9888-a26fdd1055cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws\" (UID: \"bf5c08e1-364b-456a-9888-a26fdd1055cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:00.510152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.510057 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hxb\" (UniqueName: \"kubernetes.io/projected/bf5c08e1-364b-456a-9888-a26fdd1055cb-kube-api-access-j4hxb\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws\" (UID: \"bf5c08e1-364b-456a-9888-a26fdd1055cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:00.510152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.510089 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e566451a-09cf-4392-9006-23a6444553d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq\" (UID: \"e566451a-09cf-4392-9006-23a6444553d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:00.510152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.510130 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mphs\" (UniqueName: \"kubernetes.io/projected/feee7683-a6bb-47fb-b085-6a510fc1b993-kube-api-access-6mphs\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952\" (UID: \"feee7683-a6bb-47fb-b085-6a510fc1b993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:00.510319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.510296 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4wf\" (UniqueName: \"kubernetes.io/projected/a0041a35-bbe1-40b1-ac40-f28760c6ec17-kube-api-access-gf4wf\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w\" (UID: \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:00.510377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.510348 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5076f0fd-cf19-46e1-8cb5-ab933b424acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl\" (UID: \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:00.510461 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.510443 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gbrt\" (UniqueName: \"kubernetes.io/projected/5076f0fd-cf19-46e1-8cb5-ab933b424acb-kube-api-access-2gbrt\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl\" (UID: \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:00.510516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.510501 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feee7683-a6bb-47fb-b085-6a510fc1b993-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952\" (UID: \"feee7683-a6bb-47fb-b085-6a510fc1b993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:00.511424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.511397 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feee7683-a6bb-47fb-b085-6a510fc1b993-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952\" (UID: \"feee7683-a6bb-47fb-b085-6a510fc1b993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:00.512147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.512125 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5c08e1-364b-456a-9888-a26fdd1055cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws\" (UID: \"bf5c08e1-364b-456a-9888-a26fdd1055cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:00.513357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.513331 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5076f0fd-cf19-46e1-8cb5-ab933b424acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl\" (UID: \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:00.515533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.514561 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqmv\" (UniqueName: \"kubernetes.io/projected/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-kube-api-access-zgqmv\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb\" (UID: \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:00.520734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.520710 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8"] Oct 30 14:45:00.520828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.520817 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:00.522442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.522423 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9"] Oct 30 14:45:00.540003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.539978 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hxb\" (UniqueName: \"kubernetes.io/projected/bf5c08e1-364b-456a-9888-a26fdd1055cb-kube-api-access-j4hxb\") pod \"maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws\" (UID: \"bf5c08e1-364b-456a-9888-a26fdd1055cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:00.547476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.540749 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mphs\" (UniqueName: \"kubernetes.io/projected/feee7683-a6bb-47fb-b085-6a510fc1b993-kube-api-access-6mphs\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952\" (UID: \"feee7683-a6bb-47fb-b085-6a510fc1b993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:00.547476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.541932 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gbrt\" (UniqueName: \"kubernetes.io/projected/5076f0fd-cf19-46e1-8cb5-ab933b424acb-kube-api-access-2gbrt\") pod \"maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl\" (UID: \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:00.548122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.548092 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:00.585842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.585809 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:00.603229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.601141 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:00.612313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.612293 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhnm5\" (UniqueName: \"kubernetes.io/projected/3069b2d7-58d3-4095-ae46-c2e8d355931a-kube-api-access-zhnm5\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8\" (UID: \"3069b2d7-58d3-4095-ae46-c2e8d355931a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:00.612726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.612708 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069b2d7-58d3-4095-ae46-c2e8d355931a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8\" (UID: \"3069b2d7-58d3-4095-ae46-c2e8d355931a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:00.613338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.613315 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4k5\" (UniqueName: \"kubernetes.io/projected/e566451a-09cf-4392-9006-23a6444553d9-kube-api-access-sf4k5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq\" (UID: \"e566451a-09cf-4392-9006-23a6444553d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:00.613501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.613484 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0041a35-bbe1-40b1-ac40-f28760c6ec17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w\" (UID: \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:00.614123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.614107 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e566451a-09cf-4392-9006-23a6444553d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq\" (UID: \"e566451a-09cf-4392-9006-23a6444553d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:00.614361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.614034 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0041a35-bbe1-40b1-ac40-f28760c6ec17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w\" (UID: \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:00.614361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.613386 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069b2d7-58d3-4095-ae46-c2e8d355931a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8\" (UID: \"3069b2d7-58d3-4095-ae46-c2e8d355931a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:00.614639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.614617 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4wf\" (UniqueName: \"kubernetes.io/projected/a0041a35-bbe1-40b1-ac40-f28760c6ec17-kube-api-access-gf4wf\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w\" (UID: \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:00.614953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.614933 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9\" (UID: \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:00.615306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.615288 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45v52\" (UniqueName: \"kubernetes.io/projected/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-kube-api-access-45v52\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9\" (UID: \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:00.615430 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.614784 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e566451a-09cf-4392-9006-23a6444553d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq\" (UID: \"e566451a-09cf-4392-9006-23a6444553d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:00.617098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.616926 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:00.711284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.648574 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4wf\" (UniqueName: \"kubernetes.io/projected/a0041a35-bbe1-40b1-ac40-f28760c6ec17-kube-api-access-gf4wf\") pod \"maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w\" (UID: \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:00.711284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.655599 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:00.711284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.667735 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4k5\" (UniqueName: \"kubernetes.io/projected/e566451a-09cf-4392-9006-23a6444553d9-kube-api-access-sf4k5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq\" (UID: \"e566451a-09cf-4392-9006-23a6444553d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:00.711284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.668805 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhnm5\" (UniqueName: \"kubernetes.io/projected/3069b2d7-58d3-4095-ae46-c2e8d355931a-kube-api-access-zhnm5\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8\" (UID: \"3069b2d7-58d3-4095-ae46-c2e8d355931a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:00.711284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.687723 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:00.711284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.696487 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:00.717013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.716984 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45v52\" (UniqueName: \"kubernetes.io/projected/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-kube-api-access-45v52\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9\" (UID: \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:00.717843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.717820 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9\" (UID: \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:00.718801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.718683 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9\" (UID: \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:00.729758 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.728147 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:00.773992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.773923 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45v52\" (UniqueName: \"kubernetes.io/projected/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-kube-api-access-45v52\") pod \"maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9\" (UID: \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:00.854506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.854280 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:00.874555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.874522 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:00.918874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.912623 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:00.918874 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:00.918643 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:01.102403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.102367 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx"] Oct 30 14:45:01.200542 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:01.200486 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6770fcff_79f8_4f91_a951_308c8ed76836.slice/crio-1355fa676f27d7900e0c1b3f08c653f712d2104acfc68ceac6610b51e107c14c WatchSource:0}: Error finding container 1355fa676f27d7900e0c1b3f08c653f712d2104acfc68ceac6610b51e107c14c: Status 404 returned error can't find the container with id 1355fa676f27d7900e0c1b3f08c653f712d2104acfc68ceac6610b51e107c14c Oct 30 14:45:01.292779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.292739 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" event={"ID":"6770fcff-79f8-4f91-a951-308c8ed76836","Type":"ContainerStarted","Data":"1355fa676f27d7900e0c1b3f08c653f712d2104acfc68ceac6610b51e107c14c"} Oct 30 14:45:01.371557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.371528 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb"] Oct 30 14:45:01.411032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.410995 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws"] Oct 30 14:45:01.451374 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.451339 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf"] Oct 30 14:45:01.503976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.503935 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh"] Oct 30 14:45:01.656250 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:01.649089 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acc1072_69bf_4a93_8428_90ba63f5dea8.slice/crio-4246b75c4b630bd5cbdec45711b6ef7adcc0f4fcf830b22ffb9217ea67fa22ed WatchSource:0}: Error finding container 4246b75c4b630bd5cbdec45711b6ef7adcc0f4fcf830b22ffb9217ea67fa22ed: Status 404 returned error can't find the container with id 4246b75c4b630bd5cbdec45711b6ef7adcc0f4fcf830b22ffb9217ea67fa22ed Oct 30 14:45:01.736908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.736875 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws"] Oct 30 14:45:01.747424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.747326 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952"] Oct 30 14:45:01.974873 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.974805 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl"] Oct 30 14:45:01.982092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.982065 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w"] Oct 30 14:45:01.992292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.992178 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq"] Oct 30 14:45:02.002718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:01.998773 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9"] Oct 30 14:45:02.021054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.021023 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8"] Oct 30 14:45:02.041298 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:02.041145 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3069b2d7_58d3_4095_ae46_c2e8d355931a.slice/crio-c3145e04e1226d9a4917e4c4913e16e48ee1075576d15378e04adf88f9fd37ce WatchSource:0}: Error finding container c3145e04e1226d9a4917e4c4913e16e48ee1075576d15378e04adf88f9fd37ce: Status 404 returned error can't find the container with id c3145e04e1226d9a4917e4c4913e16e48ee1075576d15378e04adf88f9fd37ce Oct 30 14:45:02.304585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.304543 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" event={"ID":"7acc1072-69bf-4a93-8428-90ba63f5dea8","Type":"ContainerStarted","Data":"4246b75c4b630bd5cbdec45711b6ef7adcc0f4fcf830b22ffb9217ea67fa22ed"} Oct 30 14:45:02.306205 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.306167 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" event={"ID":"a0041a35-bbe1-40b1-ac40-f28760c6ec17","Type":"ContainerStarted","Data":"432cf5ad9406495afd060dd4d17c0a8bcf36385a0cd09304b7636d63b0ac52a6"} Oct 30 14:45:02.308201 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.308153 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" event={"ID":"3661e6e3-d3ff-4e1d-a171-cd8ab00132be","Type":"ContainerStarted","Data":"8248621f9a92cfa41bfb56ba67097e5f01876f9cad442f883af322b9b4881e87"} Oct 30 14:45:02.308343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.308253 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" event={"ID":"3661e6e3-d3ff-4e1d-a171-cd8ab00132be","Type":"ContainerStarted","Data":"92a402bfdc58f6bc4b6a95fe47b8fb310dd6f2735563d001af57efd3e03af3c0"} Oct 30 14:45:02.321590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.321563 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" event={"ID":"e6cc8620-529c-47e4-ad22-0d4f8e998bc0","Type":"ContainerStarted","Data":"981d5a3a83277346646314434166fe26194eef9ed2bd378c3583e200808a715e"} Oct 30 14:45:02.321715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.321605 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" event={"ID":"6770fcff-79f8-4f91-a951-308c8ed76836","Type":"ContainerStarted","Data":"2a14d368786cf494a859058b2fec0ba1594f3a67b7556cedec340f482bfc5e92"} Oct 30 14:45:02.321715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.321628 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" event={"ID":"feee7683-a6bb-47fb-b085-6a510fc1b993","Type":"ContainerStarted","Data":"34574efb78fe558798a410fcb553bf610d95f881f73dc3f01460f746cbc4bf99"} Oct 30 14:45:02.321715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.321648 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" event={"ID":"5076f0fd-cf19-46e1-8cb5-ab933b424acb","Type":"ContainerStarted","Data":"8614fc4f6a792eaee88017cb44382f76655f031e215f66d2d99fd471e6e3af61"} Oct 30 14:45:02.321715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.321664 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" event={"ID":"bf5c08e1-364b-456a-9888-a26fdd1055cb","Type":"ContainerStarted","Data":"6e466bf5ca736203e31e35e1a766192f8e1ffb6e4a91385de5999e8b9ae7050e"} Oct 30 14:45:02.321715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.321684 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" event={"ID":"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e","Type":"ContainerStarted","Data":"ffe78aedaf307bf68827963dd05a5889b02599ff8dbb5fb68cc4677faecb50a9"} Oct 30 14:45:02.321715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.321708 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" event={"ID":"3069b2d7-58d3-4095-ae46-c2e8d355931a","Type":"ContainerStarted","Data":"c3145e04e1226d9a4917e4c4913e16e48ee1075576d15378e04adf88f9fd37ce"} Oct 30 14:45:02.321999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.321730 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" event={"ID":"e566451a-09cf-4392-9006-23a6444553d9","Type":"ContainerStarted","Data":"c17673dfc46e4c2bf9bbee593e95892c5f3dc9b4428ab031ff875ccf12f47289"} Oct 30 14:45:02.322687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.322662 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" event={"ID":"110d3615-218b-49be-b6b6-99062913a1df","Type":"ContainerStarted","Data":"64fd98e87ecba83afb1b4ca0250a9c03e4a7f536805a20fdd908643b71ffc2e4"} Oct 30 14:45:02.356591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.356548 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" podStartSLOduration=2.356533492 podStartE2EDuration="2.356533492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:02.353605593 +0000 UTC m=+3332.154111036" watchObservedRunningTime="2026-10-30 14:45:02.356533492 +0000 UTC m=+3332.157038947" Oct 30 14:45:02.400656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:02.400595 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" podStartSLOduration=2.400576883 podStartE2EDuration="2.400576883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:02.39891139 +0000 UTC m=+3332.199416833" watchObservedRunningTime="2026-10-30 14:45:02.400576883 +0000 UTC m=+3332.201082325" Oct 30 14:45:03.333052 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.332965 5824 generic.go:356] "Generic (PLEG): container finished" podID="110d3615-218b-49be-b6b6-99062913a1df" containerID="489d62c138f83a2b38a7664109796e171d2369b3d97528dc5708c74dc62d9fe2" exitCode=60 Oct 30 14:45:03.333548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.333039 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" event={"ID":"110d3615-218b-49be-b6b6-99062913a1df","Type":"ContainerDied","Data":"489d62c138f83a2b38a7664109796e171d2369b3d97528dc5708c74dc62d9fe2"} Oct 30 14:45:03.333548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.333294 5824 scope.go:122] "RemoveContainer" containerID="489d62c138f83a2b38a7664109796e171d2369b3d97528dc5708c74dc62d9fe2" Oct 30 14:45:03.335080 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.335055 5824 generic.go:356] "Generic (PLEG): container finished" podID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerID="7023badde8d60edefb4a19d47cd33446cf0e29042b5e6601cd3362bfad202763" exitCode=60 Oct 30 14:45:03.335229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.335120 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" event={"ID":"7acc1072-69bf-4a93-8428-90ba63f5dea8","Type":"ContainerDied","Data":"7023badde8d60edefb4a19d47cd33446cf0e29042b5e6601cd3362bfad202763"} Oct 30 14:45:03.335387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.335373 5824 scope.go:122] "RemoveContainer" containerID="7023badde8d60edefb4a19d47cd33446cf0e29042b5e6601cd3362bfad202763" Oct 30 14:45:03.337028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.337006 5824 generic.go:356] "Generic (PLEG): container finished" podID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerID="8248621f9a92cfa41bfb56ba67097e5f01876f9cad442f883af322b9b4881e87" exitCode=60 Oct 30 14:45:03.337121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.337087 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" event={"ID":"3661e6e3-d3ff-4e1d-a171-cd8ab00132be","Type":"ContainerDied","Data":"8248621f9a92cfa41bfb56ba67097e5f01876f9cad442f883af322b9b4881e87"} Oct 30 14:45:03.337373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.337312 5824 scope.go:122] "RemoveContainer" containerID="8248621f9a92cfa41bfb56ba67097e5f01876f9cad442f883af322b9b4881e87" Oct 30 14:45:03.339889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.339864 5824 generic.go:356] "Generic (PLEG): container finished" podID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerID="459ff9d387fae6ab6db5eae0d7021a952aa54a52d13ac3fb4d3706393992d54a" exitCode=60 Oct 30 14:45:03.339987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.339945 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" event={"ID":"e6cc8620-529c-47e4-ad22-0d4f8e998bc0","Type":"ContainerDied","Data":"459ff9d387fae6ab6db5eae0d7021a952aa54a52d13ac3fb4d3706393992d54a"} Oct 30 14:45:03.340141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.340124 5824 scope.go:122] "RemoveContainer" containerID="459ff9d387fae6ab6db5eae0d7021a952aa54a52d13ac3fb4d3706393992d54a" Oct 30 14:45:03.341641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.341618 5824 generic.go:356] "Generic (PLEG): container finished" podID="6770fcff-79f8-4f91-a951-308c8ed76836" containerID="2a14d368786cf494a859058b2fec0ba1594f3a67b7556cedec340f482bfc5e92" exitCode=60 Oct 30 14:45:03.341779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.341698 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" event={"ID":"6770fcff-79f8-4f91-a951-308c8ed76836","Type":"ContainerDied","Data":"2a14d368786cf494a859058b2fec0ba1594f3a67b7556cedec340f482bfc5e92"} Oct 30 14:45:03.341976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.341956 5824 scope.go:122] "RemoveContainer" containerID="2a14d368786cf494a859058b2fec0ba1594f3a67b7556cedec340f482bfc5e92" Oct 30 14:45:03.343681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.343658 5824 generic.go:356] "Generic (PLEG): container finished" podID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerID="5b5509aab785023b1f6e2993e099736636c16d16f67fe1404bd95682f799bb75" exitCode=60 Oct 30 14:45:03.343808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.343744 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" event={"ID":"feee7683-a6bb-47fb-b085-6a510fc1b993","Type":"ContainerDied","Data":"5b5509aab785023b1f6e2993e099736636c16d16f67fe1404bd95682f799bb75"} Oct 30 14:45:03.343963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.343945 5824 scope.go:122] "RemoveContainer" containerID="5b5509aab785023b1f6e2993e099736636c16d16f67fe1404bd95682f799bb75" Oct 30 14:45:03.358933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.357820 5824 generic.go:356] "Generic (PLEG): container finished" podID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerID="f20ee2e266d9ebee134fd9a1e33a8d7ec55c189822819b4488111a514eeccb04" exitCode=60 Oct 30 14:45:03.358933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.357872 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" event={"ID":"bf5c08e1-364b-456a-9888-a26fdd1055cb","Type":"ContainerDied","Data":"f20ee2e266d9ebee134fd9a1e33a8d7ec55c189822819b4488111a514eeccb04"} Oct 30 14:45:03.358933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:03.358206 5824 scope.go:122] "RemoveContainer" containerID="f20ee2e266d9ebee134fd9a1e33a8d7ec55c189822819b4488111a514eeccb04" Oct 30 14:45:03.655315 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:03.655263 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:45:04.370553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.370484 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" event={"ID":"6770fcff-79f8-4f91-a951-308c8ed76836","Type":"ContainerStarted","Data":"f5ff391bc4f2d3d7c3f8108166ce1e610de8b57f3f035f3d6bbdf69d04701d65"} Oct 30 14:45:04.373046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.373016 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" event={"ID":"feee7683-a6bb-47fb-b085-6a510fc1b993","Type":"ContainerStarted","Data":"677a8ffd252a1733ddf2ce5ff1def506fa62507842df75f3f033c27eb4adad5f"} Oct 30 14:45:04.374605 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.374583 5824 generic.go:356] "Generic (PLEG): container finished" podID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerID="fe690ea684f19790452f05858e327f00fa3f7e3bb321089410929acc58e5faaf" exitCode=60 Oct 30 14:45:04.374707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.374661 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" event={"ID":"5076f0fd-cf19-46e1-8cb5-ab933b424acb","Type":"ContainerDied","Data":"fe690ea684f19790452f05858e327f00fa3f7e3bb321089410929acc58e5faaf"} Oct 30 14:45:04.374824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.374810 5824 scope.go:122] "RemoveContainer" containerID="fe690ea684f19790452f05858e327f00fa3f7e3bb321089410929acc58e5faaf" Oct 30 14:45:04.376550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.376484 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" event={"ID":"bf5c08e1-364b-456a-9888-a26fdd1055cb","Type":"ContainerStarted","Data":"1c0820911d18800dd3ec11e6986d989fd650c97f41a6ee688d1baa4b17c02453"} Oct 30 14:45:04.378166 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.378144 5824 generic.go:356] "Generic (PLEG): container finished" podID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerID="c4e7af67764d4ea7f9979d3b7cede12c1322e538e66ffcdb9ce99cfe5b7d2b68" exitCode=60 Oct 30 14:45:04.378335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.378218 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" event={"ID":"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e","Type":"ContainerDied","Data":"c4e7af67764d4ea7f9979d3b7cede12c1322e538e66ffcdb9ce99cfe5b7d2b68"} Oct 30 14:45:04.378462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.378441 5824 scope.go:122] "RemoveContainer" containerID="c4e7af67764d4ea7f9979d3b7cede12c1322e538e66ffcdb9ce99cfe5b7d2b68" Oct 30 14:45:04.379851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.379831 5824 generic.go:356] "Generic (PLEG): container finished" podID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerID="200d4a50917a42ee15cdeaee3b5cf477afb38b2b5bbd996eb9136755bab8047f" exitCode=60 Oct 30 14:45:04.380092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.380077 5824 scope.go:122] "RemoveContainer" containerID="200d4a50917a42ee15cdeaee3b5cf477afb38b2b5bbd996eb9136755bab8047f" Oct 30 14:45:04.380229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.380132 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" event={"ID":"3069b2d7-58d3-4095-ae46-c2e8d355931a","Type":"ContainerDied","Data":"200d4a50917a42ee15cdeaee3b5cf477afb38b2b5bbd996eb9136755bab8047f"} Oct 30 14:45:04.381511 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.381494 5824 generic.go:356] "Generic (PLEG): container finished" podID="e566451a-09cf-4392-9006-23a6444553d9" containerID="169a0291357d6350435674fd882e06c9e40a1e84532136dc6dbfcee40ef2f379" exitCode=60 Oct 30 14:45:04.381597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.381577 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" event={"ID":"e566451a-09cf-4392-9006-23a6444553d9","Type":"ContainerDied","Data":"169a0291357d6350435674fd882e06c9e40a1e84532136dc6dbfcee40ef2f379"} Oct 30 14:45:04.381808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.381791 5824 scope.go:122] "RemoveContainer" containerID="169a0291357d6350435674fd882e06c9e40a1e84532136dc6dbfcee40ef2f379" Oct 30 14:45:04.383455 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.383432 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" event={"ID":"110d3615-218b-49be-b6b6-99062913a1df","Type":"ContainerStarted","Data":"3e9f49b254e2bbb48fc83cc0ef5e80603418bddd61132e0cff0938643f5e8654"} Oct 30 14:45:04.385322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.385298 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" event={"ID":"7acc1072-69bf-4a93-8428-90ba63f5dea8","Type":"ContainerStarted","Data":"8eda928c71dd2a8ae9b8099c4eaacf867346920c22669efe0d07455d74316157"} Oct 30 14:45:04.389474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.389449 5824 generic.go:356] "Generic (PLEG): container finished" podID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerID="bec884f4089ba3a50701566b035b4b80ca84c39bea01e0bdf7e2101833098212" exitCode=60 Oct 30 14:45:04.390344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.389783 5824 scope.go:122] "RemoveContainer" containerID="bec884f4089ba3a50701566b035b4b80ca84c39bea01e0bdf7e2101833098212" Oct 30 14:45:04.390344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.389791 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" event={"ID":"a0041a35-bbe1-40b1-ac40-f28760c6ec17","Type":"ContainerDied","Data":"bec884f4089ba3a50701566b035b4b80ca84c39bea01e0bdf7e2101833098212"} Oct 30 14:45:04.399104 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.399077 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" event={"ID":"3661e6e3-d3ff-4e1d-a171-cd8ab00132be","Type":"ContainerStarted","Data":"deaca9f075dc69511d5838df07cf985da7f0b838c3f9019e6bb834fc889d0a30"} Oct 30 14:45:04.401156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.401096 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" event={"ID":"e6cc8620-529c-47e4-ad22-0d4f8e998bc0","Type":"ContainerStarted","Data":"f430ea1f938f6fd8c67ce7b20151270dda85a1f1ab8f095eebee516c050587d9"} Oct 30 14:45:04.459802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.459760 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" podStartSLOduration=4.459747298 podStartE2EDuration="4.459747298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:04.458587241 +0000 UTC m=+3334.259092684" watchObservedRunningTime="2026-10-30 14:45:04.459747298 +0000 UTC m=+3334.260252774" Oct 30 14:45:04.554997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.554940 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" podStartSLOduration=4.554920864 podStartE2EDuration="4.554920864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:04.54864944 +0000 UTC m=+3334.349154883" watchObservedRunningTime="2026-10-30 14:45:04.554920864 +0000 UTC m=+3334.355426308" Oct 30 14:45:04.595169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.595109 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" podStartSLOduration=4.595094735 podStartE2EDuration="4.595094735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:04.592391329 +0000 UTC m=+3334.392896772" watchObservedRunningTime="2026-10-30 14:45:04.595094735 +0000 UTC m=+3334.395600177" Oct 30 14:45:04.685987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.685940 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" podStartSLOduration=4.685925602 podStartE2EDuration="4.685925602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:04.683221008 +0000 UTC m=+3334.483726451" watchObservedRunningTime="2026-10-30 14:45:04.685925602 +0000 UTC m=+3334.486431044" Oct 30 14:45:04.819844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:04.819796 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" podStartSLOduration=4.819782149 podStartE2EDuration="4.819782149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:04.816302885 +0000 UTC m=+3334.616808355" watchObservedRunningTime="2026-10-30 14:45:04.819782149 +0000 UTC m=+3334.620287591" Oct 30 14:45:05.411057 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.411030 5824 generic.go:356] "Generic (PLEG): container finished" podID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerID="8eda928c71dd2a8ae9b8099c4eaacf867346920c22669efe0d07455d74316157" exitCode=60 Oct 30 14:45:05.411504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.411093 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" event={"ID":"7acc1072-69bf-4a93-8428-90ba63f5dea8","Type":"ContainerDied","Data":"8eda928c71dd2a8ae9b8099c4eaacf867346920c22669efe0d07455d74316157"} Oct 30 14:45:05.411504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.411168 5824 scope.go:122] "RemoveContainer" containerID="7023badde8d60edefb4a19d47cd33446cf0e29042b5e6601cd3362bfad202763" Oct 30 14:45:05.411504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.411390 5824 scope.go:122] "RemoveContainer" containerID="8eda928c71dd2a8ae9b8099c4eaacf867346920c22669efe0d07455d74316157" Oct 30 14:45:05.411713 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:05.411691 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh_odh-ai-gateway-infra(7acc1072-69bf-4a93-8428-90ba63f5dea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" Oct 30 14:45:05.413441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.413418 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" event={"ID":"a0041a35-bbe1-40b1-ac40-f28760c6ec17","Type":"ContainerStarted","Data":"d3046c10fd09e90d8a908b67462cd2b20287128d0f0d3481f064816f991af878"} Oct 30 14:45:05.415556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.415538 5824 generic.go:356] "Generic (PLEG): container finished" podID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerID="deaca9f075dc69511d5838df07cf985da7f0b838c3f9019e6bb834fc889d0a30" exitCode=60 Oct 30 14:45:05.415654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.415608 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" event={"ID":"3661e6e3-d3ff-4e1d-a171-cd8ab00132be","Type":"ContainerDied","Data":"deaca9f075dc69511d5838df07cf985da7f0b838c3f9019e6bb834fc889d0a30"} Oct 30 14:45:05.415847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.415822 5824 scope.go:122] "RemoveContainer" containerID="deaca9f075dc69511d5838df07cf985da7f0b838c3f9019e6bb834fc889d0a30" Oct 30 14:45:05.416102 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:05.416066 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb_odh-ai-gateway-infra(3661e6e3-d3ff-4e1d-a171-cd8ab00132be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" Oct 30 14:45:05.417819 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.417799 5824 generic.go:356] "Generic (PLEG): container finished" podID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerID="f430ea1f938f6fd8c67ce7b20151270dda85a1f1ab8f095eebee516c050587d9" exitCode=60 Oct 30 14:45:05.417917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.417864 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" event={"ID":"e6cc8620-529c-47e4-ad22-0d4f8e998bc0","Type":"ContainerDied","Data":"f430ea1f938f6fd8c67ce7b20151270dda85a1f1ab8f095eebee516c050587d9"} Oct 30 14:45:05.418076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.418058 5824 scope.go:122] "RemoveContainer" containerID="f430ea1f938f6fd8c67ce7b20151270dda85a1f1ab8f095eebee516c050587d9" Oct 30 14:45:05.418318 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:05.418299 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws_odh-ai-gateway-infra(e6cc8620-529c-47e4-ad22-0d4f8e998bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" Oct 30 14:45:05.420051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.420030 5824 generic.go:356] "Generic (PLEG): container finished" podID="6770fcff-79f8-4f91-a951-308c8ed76836" containerID="f5ff391bc4f2d3d7c3f8108166ce1e610de8b57f3f035f3d6bbdf69d04701d65" exitCode=60 Oct 30 14:45:05.420142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.420107 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" event={"ID":"6770fcff-79f8-4f91-a951-308c8ed76836","Type":"ContainerDied","Data":"f5ff391bc4f2d3d7c3f8108166ce1e610de8b57f3f035f3d6bbdf69d04701d65"} Oct 30 14:45:05.420316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.420302 5824 scope.go:122] "RemoveContainer" containerID="f5ff391bc4f2d3d7c3f8108166ce1e610de8b57f3f035f3d6bbdf69d04701d65" Oct 30 14:45:05.420530 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:05.420513 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx_odh-ai-gateway-infra(6770fcff-79f8-4f91-a951-308c8ed76836)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" Oct 30 14:45:05.422152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.422134 5824 generic.go:356] "Generic (PLEG): container finished" podID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerID="677a8ffd252a1733ddf2ce5ff1def506fa62507842df75f3f033c27eb4adad5f" exitCode=60 Oct 30 14:45:05.422268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.422241 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" event={"ID":"feee7683-a6bb-47fb-b085-6a510fc1b993","Type":"ContainerDied","Data":"677a8ffd252a1733ddf2ce5ff1def506fa62507842df75f3f033c27eb4adad5f"} Oct 30 14:45:05.422446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.422430 5824 scope.go:122] "RemoveContainer" containerID="677a8ffd252a1733ddf2ce5ff1def506fa62507842df75f3f033c27eb4adad5f" Oct 30 14:45:05.422702 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:05.422675 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952_odh-ai-gateway-infra(feee7683-a6bb-47fb-b085-6a510fc1b993)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" Oct 30 14:45:05.424465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.424434 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" event={"ID":"5076f0fd-cf19-46e1-8cb5-ab933b424acb","Type":"ContainerStarted","Data":"b63c93c8f2a05f66f022d65fe2e1f31ab81775953c0a1429380ee5eeeeb6e382"} Oct 30 14:45:05.426198 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.426167 5824 scope.go:122] "RemoveContainer" containerID="8248621f9a92cfa41bfb56ba67097e5f01876f9cad442f883af322b9b4881e87" Oct 30 14:45:05.426479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.426454 5824 generic.go:356] "Generic (PLEG): container finished" podID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerID="1c0820911d18800dd3ec11e6986d989fd650c97f41a6ee688d1baa4b17c02453" exitCode=60 Oct 30 14:45:05.426569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.426527 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" event={"ID":"bf5c08e1-364b-456a-9888-a26fdd1055cb","Type":"ContainerDied","Data":"1c0820911d18800dd3ec11e6986d989fd650c97f41a6ee688d1baa4b17c02453"} Oct 30 14:45:05.426745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.426728 5824 scope.go:122] "RemoveContainer" containerID="1c0820911d18800dd3ec11e6986d989fd650c97f41a6ee688d1baa4b17c02453" Oct 30 14:45:05.427004 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:05.426972 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws_odh-ai-gateway-infra(bf5c08e1-364b-456a-9888-a26fdd1055cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" Oct 30 14:45:05.428582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.428559 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" event={"ID":"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e","Type":"ContainerStarted","Data":"0e352f359fb1157a8c798c9be3651191c60a8e789033a65f1e2541d986155912"} Oct 30 14:45:05.430414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.430394 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" event={"ID":"3069b2d7-58d3-4095-ae46-c2e8d355931a","Type":"ContainerStarted","Data":"cb2382b8661bc15794cd50d314975634423ebc025821ab3ff8756ad00faa2a48"} Oct 30 14:45:05.432386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.432356 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" event={"ID":"e566451a-09cf-4392-9006-23a6444553d9","Type":"ContainerStarted","Data":"4b6d91a53bc02e9edceb4c04ae0ce6a3990ce62a6b00c9186c68bd8b22a52d75"} Oct 30 14:45:05.434256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.434232 5824 generic.go:356] "Generic (PLEG): container finished" podID="110d3615-218b-49be-b6b6-99062913a1df" containerID="3e9f49b254e2bbb48fc83cc0ef5e80603418bddd61132e0cff0938643f5e8654" exitCode=60 Oct 30 14:45:05.434342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.434279 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" event={"ID":"110d3615-218b-49be-b6b6-99062913a1df","Type":"ContainerDied","Data":"3e9f49b254e2bbb48fc83cc0ef5e80603418bddd61132e0cff0938643f5e8654"} Oct 30 14:45:05.434501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.434489 5824 scope.go:122] "RemoveContainer" containerID="3e9f49b254e2bbb48fc83cc0ef5e80603418bddd61132e0cff0938643f5e8654" Oct 30 14:45:05.434682 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:05.434667 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf_odh-ai-gateway-infra(110d3615-218b-49be-b6b6-99062913a1df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" podUID="110d3615-218b-49be-b6b6-99062913a1df" Oct 30 14:45:05.506478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.506403 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" podStartSLOduration=5.506389434 podStartE2EDuration="5.506389434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:05.502308659 +0000 UTC m=+3335.302814139" watchObservedRunningTime="2026-10-30 14:45:05.506389434 +0000 UTC m=+3335.306894897" Oct 30 14:45:05.549446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.549395 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" podStartSLOduration=5.549381923 podStartE2EDuration="5.549381923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:05.548363996 +0000 UTC m=+3335.348869441" watchObservedRunningTime="2026-10-30 14:45:05.549381923 +0000 UTC m=+3335.349887364" Oct 30 14:45:05.587659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.587636 5824 scope.go:122] "RemoveContainer" containerID="459ff9d387fae6ab6db5eae0d7021a952aa54a52d13ac3fb4d3706393992d54a" Oct 30 14:45:05.602464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.602439 5824 scope.go:122] "RemoveContainer" containerID="2a14d368786cf494a859058b2fec0ba1594f3a67b7556cedec340f482bfc5e92" Oct 30 14:45:05.615709 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.615683 5824 scope.go:122] "RemoveContainer" containerID="5b5509aab785023b1f6e2993e099736636c16d16f67fe1404bd95682f799bb75" Oct 30 14:45:05.629110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.629081 5824 scope.go:122] "RemoveContainer" containerID="f20ee2e266d9ebee134fd9a1e33a8d7ec55c189822819b4488111a514eeccb04" Oct 30 14:45:05.643571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.643535 5824 scope.go:122] "RemoveContainer" containerID="489d62c138f83a2b38a7664109796e171d2369b3d97528dc5708c74dc62d9fe2" Oct 30 14:45:05.844656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.844603 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" podStartSLOduration=5.844588148 podStartE2EDuration="5.844588148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:05.840869219 +0000 UTC m=+3335.641374662" watchObservedRunningTime="2026-10-30 14:45:05.844588148 +0000 UTC m=+3335.645093606" Oct 30 14:45:05.904267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.904176 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" podStartSLOduration=5.904162879 podStartE2EDuration="5.904162879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:05.899400474 +0000 UTC m=+3335.699905917" watchObservedRunningTime="2026-10-30 14:45:05.904162879 +0000 UTC m=+3335.704668321" Oct 30 14:45:05.972168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:05.972115 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" podStartSLOduration=5.972101001 podStartE2EDuration="5.972101001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:05.969274137 +0000 UTC m=+3335.769779575" watchObservedRunningTime="2026-10-30 14:45:05.972101001 +0000 UTC m=+3335.772606444" Oct 30 14:45:06.062005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.061912 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf"] Oct 30 14:45:06.075527 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.075494 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz"] Oct 30 14:45:06.444833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.444801 5824 generic.go:356] "Generic (PLEG): container finished" podID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerID="cb2382b8661bc15794cd50d314975634423ebc025821ab3ff8756ad00faa2a48" exitCode=60 Oct 30 14:45:06.445319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.444880 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" event={"ID":"3069b2d7-58d3-4095-ae46-c2e8d355931a","Type":"ContainerDied","Data":"cb2382b8661bc15794cd50d314975634423ebc025821ab3ff8756ad00faa2a48"} Oct 30 14:45:06.445319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.444945 5824 scope.go:122] "RemoveContainer" containerID="200d4a50917a42ee15cdeaee3b5cf477afb38b2b5bbd996eb9136755bab8047f" Oct 30 14:45:06.445319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.445171 5824 scope.go:122] "RemoveContainer" containerID="cb2382b8661bc15794cd50d314975634423ebc025821ab3ff8756ad00faa2a48" Oct 30 14:45:06.445595 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.445569 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8_odh-ai-gateway-infra(3069b2d7-58d3-4095-ae46-c2e8d355931a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" Oct 30 14:45:06.447478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.447451 5824 generic.go:356] "Generic (PLEG): container finished" podID="e566451a-09cf-4392-9006-23a6444553d9" containerID="4b6d91a53bc02e9edceb4c04ae0ce6a3990ce62a6b00c9186c68bd8b22a52d75" exitCode=60 Oct 30 14:45:06.447607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.447478 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" event={"ID":"e566451a-09cf-4392-9006-23a6444553d9","Type":"ContainerDied","Data":"4b6d91a53bc02e9edceb4c04ae0ce6a3990ce62a6b00c9186c68bd8b22a52d75"} Oct 30 14:45:06.447820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.447798 5824 scope.go:122] "RemoveContainer" containerID="4b6d91a53bc02e9edceb4c04ae0ce6a3990ce62a6b00c9186c68bd8b22a52d75" Oct 30 14:45:06.448114 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.448089 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq_odh-ai-gateway-infra(e566451a-09cf-4392-9006-23a6444553d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" podUID="e566451a-09cf-4392-9006-23a6444553d9" Oct 30 14:45:06.449816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.449796 5824 scope.go:122] "RemoveContainer" containerID="3e9f49b254e2bbb48fc83cc0ef5e80603418bddd61132e0cff0938643f5e8654" Oct 30 14:45:06.450066 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.450045 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf_odh-ai-gateway-infra(110d3615-218b-49be-b6b6-99062913a1df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" podUID="110d3615-218b-49be-b6b6-99062913a1df" Oct 30 14:45:06.451582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.451565 5824 scope.go:122] "RemoveContainer" containerID="8eda928c71dd2a8ae9b8099c4eaacf867346920c22669efe0d07455d74316157" Oct 30 14:45:06.451814 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.451797 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh_odh-ai-gateway-infra(7acc1072-69bf-4a93-8428-90ba63f5dea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" Oct 30 14:45:06.453442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.453420 5824 generic.go:356] "Generic (PLEG): container finished" podID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerID="d3046c10fd09e90d8a908b67462cd2b20287128d0f0d3481f064816f991af878" exitCode=60 Oct 30 14:45:06.453519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.453504 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" event={"ID":"a0041a35-bbe1-40b1-ac40-f28760c6ec17","Type":"ContainerDied","Data":"d3046c10fd09e90d8a908b67462cd2b20287128d0f0d3481f064816f991af878"} Oct 30 14:45:06.453707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.453688 5824 scope.go:122] "RemoveContainer" containerID="d3046c10fd09e90d8a908b67462cd2b20287128d0f0d3481f064816f991af878" Oct 30 14:45:06.453982 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.453957 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w_odh-ai-gateway-infra(a0041a35-bbe1-40b1-ac40-f28760c6ec17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" Oct 30 14:45:06.455587 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.455567 5824 scope.go:122] "RemoveContainer" containerID="deaca9f075dc69511d5838df07cf985da7f0b838c3f9019e6bb834fc889d0a30" Oct 30 14:45:06.455835 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.455812 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb_odh-ai-gateway-infra(3661e6e3-d3ff-4e1d-a171-cd8ab00132be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" Oct 30 14:45:06.457389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.457369 5824 scope.go:122] "RemoveContainer" containerID="f430ea1f938f6fd8c67ce7b20151270dda85a1f1ab8f095eebee516c050587d9" Oct 30 14:45:06.457626 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.457604 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws_odh-ai-gateway-infra(e6cc8620-529c-47e4-ad22-0d4f8e998bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" Oct 30 14:45:06.459352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.459335 5824 scope.go:122] "RemoveContainer" containerID="f5ff391bc4f2d3d7c3f8108166ce1e610de8b57f3f035f3d6bbdf69d04701d65" Oct 30 14:45:06.459567 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.459546 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx_odh-ai-gateway-infra(6770fcff-79f8-4f91-a951-308c8ed76836)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" Oct 30 14:45:06.460743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.460728 5824 scope.go:122] "RemoveContainer" containerID="169a0291357d6350435674fd882e06c9e40a1e84532136dc6dbfcee40ef2f379" Oct 30 14:45:06.461164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.461145 5824 scope.go:122] "RemoveContainer" containerID="677a8ffd252a1733ddf2ce5ff1def506fa62507842df75f3f033c27eb4adad5f" Oct 30 14:45:06.461513 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.461489 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952_odh-ai-gateway-infra(feee7683-a6bb-47fb-b085-6a510fc1b993)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" Oct 30 14:45:06.463157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.463136 5824 generic.go:356] "Generic (PLEG): container finished" podID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerID="b63c93c8f2a05f66f022d65fe2e1f31ab81775953c0a1429380ee5eeeeb6e382" exitCode=60 Oct 30 14:45:06.463259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.463236 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" event={"ID":"5076f0fd-cf19-46e1-8cb5-ab933b424acb","Type":"ContainerDied","Data":"b63c93c8f2a05f66f022d65fe2e1f31ab81775953c0a1429380ee5eeeeb6e382"} Oct 30 14:45:06.463442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.463422 5824 scope.go:122] "RemoveContainer" containerID="b63c93c8f2a05f66f022d65fe2e1f31ab81775953c0a1429380ee5eeeeb6e382" Oct 30 14:45:06.463655 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.463632 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl_odh-ai-gateway-infra(5076f0fd-cf19-46e1-8cb5-ab933b424acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" Oct 30 14:45:06.465090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.465072 5824 scope.go:122] "RemoveContainer" containerID="1c0820911d18800dd3ec11e6986d989fd650c97f41a6ee688d1baa4b17c02453" Oct 30 14:45:06.465332 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.465310 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws_odh-ai-gateway-infra(bf5c08e1-364b-456a-9888-a26fdd1055cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" Oct 30 14:45:06.466934 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.466912 5824 generic.go:356] "Generic (PLEG): container finished" podID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerID="0e352f359fb1157a8c798c9be3651191c60a8e789033a65f1e2541d986155912" exitCode=60 Oct 30 14:45:06.467026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.466971 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" event={"ID":"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e","Type":"ContainerDied","Data":"0e352f359fb1157a8c798c9be3651191c60a8e789033a65f1e2541d986155912"} Oct 30 14:45:06.467297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.467280 5824 scope.go:122] "RemoveContainer" containerID="0e352f359fb1157a8c798c9be3651191c60a8e789033a65f1e2541d986155912" Oct 30 14:45:06.467554 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:06.467534 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9_odh-ai-gateway-infra(6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" Oct 30 14:45:06.475923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.475827 5824 scope.go:122] "RemoveContainer" containerID="bec884f4089ba3a50701566b035b4b80ca84c39bea01e0bdf7e2101833098212" Oct 30 14:45:06.492634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.492610 5824 scope.go:122] "RemoveContainer" containerID="fe690ea684f19790452f05858e327f00fa3f7e3bb321089410929acc58e5faaf" Oct 30 14:45:06.517915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.517894 5824 scope.go:122] "RemoveContainer" containerID="c4e7af67764d4ea7f9979d3b7cede12c1322e538e66ffcdb9ce99cfe5b7d2b68" Oct 30 14:45:06.588726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.588700 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f"] Oct 30 14:45:06.595088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.595067 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.627897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.627867 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f"] Oct 30 14:45:06.635946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.635911 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.636072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.636014 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf9caa-0df3-4620-b305-485bb89320ef-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.636072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.636052 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.636156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.636108 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.636216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.636183 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.636279 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.636261 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtllg\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-kube-api-access-xtllg\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.636429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.636406 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.636705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.636661 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf9caa-0df3-4620-b305-485bb89320ef-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.636784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.636703 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.737656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.737549 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf9caa-0df3-4620-b305-485bb89320ef-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.737656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.737599 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.737871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.737667 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.737871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.737738 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf9caa-0df3-4620-b305-485bb89320ef-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.737871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.737768 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.738041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.737984 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.738112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.738075 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.738172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.738138 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtllg\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-kube-api-access-xtllg\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.738273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.738245 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-data\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.738342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.738275 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.738400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.738360 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-certs\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.738548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.738523 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.738961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.738942 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-credential-socket\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.739470 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.739442 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf9caa-0df3-4620-b305-485bb89320ef-istiod-ca-cert\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.740679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.740660 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-envoy\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.740762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.740750 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf9caa-0df3-4620-b305-485bb89320ef-istio-podinfo\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.765232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.763855 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-istio-token\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.765232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.764752 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtllg\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-kube-api-access-xtllg\") pod \"e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:06.914361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:06.914323 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:07.251266 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:07.251236 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bf9caa_0df3_4620_b305_485bb89320ef.slice/crio-0a824398f7d077c4426837f23aa976b67782d4890194e8a0ed9e0ca0646b51a8 WatchSource:0}: Error finding container 0a824398f7d077c4426837f23aa976b67782d4890194e8a0ed9e0ca0646b51a8: Status 404 returned error can't find the container with id 0a824398f7d077c4426837f23aa976b67782d4890194e8a0ed9e0ca0646b51a8 Oct 30 14:45:07.252863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.252835 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f"] Oct 30 14:45:07.254060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.254027 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:45:07.254143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.254094 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:45:07.254143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.254123 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:45:07.489918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.489890 5824 scope.go:122] "RemoveContainer" containerID="0e352f359fb1157a8c798c9be3651191c60a8e789033a65f1e2541d986155912" Oct 30 14:45:07.490412 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:07.490181 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9_odh-ai-gateway-infra(6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" Oct 30 14:45:07.492043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.492023 5824 scope.go:122] "RemoveContainer" containerID="cb2382b8661bc15794cd50d314975634423ebc025821ab3ff8756ad00faa2a48" Oct 30 14:45:07.492350 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:07.492325 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8_odh-ai-gateway-infra(3069b2d7-58d3-4095-ae46-c2e8d355931a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" Oct 30 14:45:07.493846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.493816 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" event={"ID":"52bf9caa-0df3-4620-b305-485bb89320ef","Type":"ContainerStarted","Data":"9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723"} Oct 30 14:45:07.493945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.493901 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" event={"ID":"52bf9caa-0df3-4620-b305-485bb89320ef","Type":"ContainerStarted","Data":"0a824398f7d077c4426837f23aa976b67782d4890194e8a0ed9e0ca0646b51a8"} Oct 30 14:45:07.496133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.496110 5824 scope.go:122] "RemoveContainer" containerID="4b6d91a53bc02e9edceb4c04ae0ce6a3990ce62a6b00c9186c68bd8b22a52d75" Oct 30 14:45:07.496448 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:07.496418 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq_odh-ai-gateway-infra(e566451a-09cf-4392-9006-23a6444553d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" podUID="e566451a-09cf-4392-9006-23a6444553d9" Oct 30 14:45:07.498074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.498025 5824 scope.go:122] "RemoveContainer" containerID="d3046c10fd09e90d8a908b67462cd2b20287128d0f0d3481f064816f991af878" Oct 30 14:45:07.498264 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:07.498246 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w_odh-ai-gateway-infra(a0041a35-bbe1-40b1-ac40-f28760c6ec17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" Oct 30 14:45:07.499774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.499755 5824 scope.go:122] "RemoveContainer" containerID="b63c93c8f2a05f66f022d65fe2e1f31ab81775953c0a1429380ee5eeeeb6e382" Oct 30 14:45:07.500028 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:07.500007 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl_odh-ai-gateway-infra(5076f0fd-cf19-46e1-8cb5-ab933b424acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" Oct 30 14:45:07.675578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.675522 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" podStartSLOduration=1.6755044369999998 podStartE2EDuration="1.675504437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:07.673909902 +0000 UTC m=+3337.474415342" watchObservedRunningTime="2026-10-30 14:45:07.675504437 +0000 UTC m=+3337.476009882" Oct 30 14:45:07.915483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:07.915403 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:08.917408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:08.917369 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:45:08.917925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:08.917426 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 14:45:09.914685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:09.914648 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:45:09.914900 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:09.914706 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 14:45:10.287384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.287351 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5"] Oct 30 14:45:10.306247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.305455 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.322895 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.322866 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5"] Oct 30 14:45:10.385020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.381854 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:45:10.405672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.405641 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvpq\" (UniqueName: \"kubernetes.io/projected/c51621a4-9cc5-4fb6-be52-903bee548a74-kube-api-access-jzvpq\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.405887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.405865 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.406011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.405995 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.507257 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.507220 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.507496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.507328 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.507577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.507521 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvpq\" (UniqueName: \"kubernetes.io/projected/c51621a4-9cc5-4fb6-be52-903bee548a74-kube-api-access-jzvpq\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.507956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.507930 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.509575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.509553 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.533564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.533534 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvpq\" (UniqueName: \"kubernetes.io/projected/c51621a4-9cc5-4fb6-be52-903bee548a74-kube-api-access-jzvpq\") pod \"maas-api-revoke-sub-bf056ee6-q65v5\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.629064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.628989 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:10.916006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.915096 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:45:10.916006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.915223 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 14:45:10.974598 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:10.974563 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5"] Oct 30 14:45:10.976136 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:10.974942 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51621a4_9cc5_4fb6_be52_903bee548a74.slice/crio-1fb631d9e5a8a0fd25bbe1ff48f4a29141b43fc06517caa9309a9d9b9b01ab3d WatchSource:0}: Error finding container 1fb631d9e5a8a0fd25bbe1ff48f4a29141b43fc06517caa9309a9d9b9b01ab3d: Status 404 returned error can't find the container with id 1fb631d9e5a8a0fd25bbe1ff48f4a29141b43fc06517caa9309a9d9b9b01ab3d Oct 30 14:45:11.068210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.067221 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" podUID="592a7ead-f353-4b52-9fb2-f2bf41af4119" containerName="payload-processing" containerID="cri-o://22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c" gracePeriod=25 Oct 30 14:45:11.078213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.077528 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" podUID="4b25e4f7-687c-429f-9395-951ebe00e027" containerName="payload-pre-processing" containerID="cri-o://58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c" gracePeriod=25 Oct 30 14:45:11.391528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.391503 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:45:11.428112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.428056 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b25e4f7-687c-429f-9395-951ebe00e027-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25e4f7-687c-429f-9395-951ebe00e027-plugins-config-volume\") pod \"4b25e4f7-687c-429f-9395-951ebe00e027\" (UID: \"4b25e4f7-687c-429f-9395-951ebe00e027\") " Oct 30 14:45:11.428112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.428094 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b25e4f7-687c-429f-9395-951ebe00e027-kube-api-access-w22x6\" (UniqueName: \"kubernetes.io/projected/4b25e4f7-687c-429f-9395-951ebe00e027-kube-api-access-w22x6\") pod \"4b25e4f7-687c-429f-9395-951ebe00e027\" (UID: \"4b25e4f7-687c-429f-9395-951ebe00e027\") " Oct 30 14:45:11.428418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.428394 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b25e4f7-687c-429f-9395-951ebe00e027-plugins-config-volume" pod "4b25e4f7-687c-429f-9395-951ebe00e027" (UID: "4b25e4f7-687c-429f-9395-951ebe00e027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:11.428852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.428828 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b25e4f7-687c-429f-9395-951ebe00e027-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:11.430382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.430362 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:45:11.430571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.430554 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b25e4f7-687c-429f-9395-951ebe00e027-kube-api-access-w22x6" pod "4b25e4f7-687c-429f-9395-951ebe00e027" (UID: "4b25e4f7-687c-429f-9395-951ebe00e027"). InnerVolumeSpecName "kube-api-access-w22x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:11.529392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.529367 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592a7ead-f353-4b52-9fb2-f2bf41af4119-kube-api-access-xsqxx\" (UniqueName: \"kubernetes.io/projected/592a7ead-f353-4b52-9fb2-f2bf41af4119-kube-api-access-xsqxx\") pod \"592a7ead-f353-4b52-9fb2-f2bf41af4119\" (UID: \"592a7ead-f353-4b52-9fb2-f2bf41af4119\") " Oct 30 14:45:11.529616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.529600 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592a7ead-f353-4b52-9fb2-f2bf41af4119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592a7ead-f353-4b52-9fb2-f2bf41af4119-plugins-config-volume\") pod \"592a7ead-f353-4b52-9fb2-f2bf41af4119\" (UID: \"592a7ead-f353-4b52-9fb2-f2bf41af4119\") " Oct 30 14:45:11.529952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.529927 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592a7ead-f353-4b52-9fb2-f2bf41af4119-plugins-config-volume" pod "592a7ead-f353-4b52-9fb2-f2bf41af4119" (UID: "592a7ead-f353-4b52-9fb2-f2bf41af4119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:11.530752 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.530668 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22x6\" (UniqueName: \"kubernetes.io/projected/4b25e4f7-687c-429f-9395-951ebe00e027-kube-api-access-w22x6\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:11.530752 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.530693 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592a7ead-f353-4b52-9fb2-f2bf41af4119-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:11.532508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.532478 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592a7ead-f353-4b52-9fb2-f2bf41af4119-kube-api-access-xsqxx" pod "592a7ead-f353-4b52-9fb2-f2bf41af4119" (UID: "592a7ead-f353-4b52-9fb2-f2bf41af4119"). InnerVolumeSpecName "kube-api-access-xsqxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:11.534826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.534803 5824 generic.go:356] "Generic (PLEG): container finished" podID="4b25e4f7-687c-429f-9395-951ebe00e027" containerID="58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c" exitCode=0 Oct 30 14:45:11.534980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.534934 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" event={"ID":"4b25e4f7-687c-429f-9395-951ebe00e027","Type":"ContainerDied","Data":"58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c"} Oct 30 14:45:11.535054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.534983 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" event={"ID":"4b25e4f7-687c-429f-9395-951ebe00e027","Type":"ContainerDied","Data":"927597a4575a2b7433c5de8f2e110de3a4a6e476cd70d093bdfbbabed2800870"} Oct 30 14:45:11.535054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.534996 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" Oct 30 14:45:11.535172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.535005 5824 scope.go:122] "RemoveContainer" containerID="58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c" Oct 30 14:45:11.538108 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.538087 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" event={"ID":"c51621a4-9cc5-4fb6-be52-903bee548a74","Type":"ContainerStarted","Data":"6f68355cd97ff88c056cb12c45f49ef90c8eee8ea12cfc6234ae0af8ae026c51"} Oct 30 14:45:11.538240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.538124 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" event={"ID":"c51621a4-9cc5-4fb6-be52-903bee548a74","Type":"ContainerStarted","Data":"1fb631d9e5a8a0fd25bbe1ff48f4a29141b43fc06517caa9309a9d9b9b01ab3d"} Oct 30 14:45:11.574295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.540093 5824 generic.go:356] "Generic (PLEG): container finished" podID="592a7ead-f353-4b52-9fb2-f2bf41af4119" containerID="22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c" exitCode=0 Oct 30 14:45:11.574295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.540151 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" event={"ID":"592a7ead-f353-4b52-9fb2-f2bf41af4119","Type":"ContainerDied","Data":"22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c"} Oct 30 14:45:11.574295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.540214 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" event={"ID":"592a7ead-f353-4b52-9fb2-f2bf41af4119","Type":"ContainerDied","Data":"d62d6c9657f23463c8e5f032eaf74c6b6e76e2e580631e936a80e0f927adb78b"} Oct 30 14:45:11.574295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.540140 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" Oct 30 14:45:11.591314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.591297 5824 scope.go:122] "RemoveContainer" containerID="58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c" Oct 30 14:45:11.591621 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:11.591599 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c\": container with ID starting with 58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c not found: ID does not exist" containerID="58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c" Oct 30 14:45:11.591721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.591626 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c"} err="failed to get container status \"58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c\": rpc error: code = NotFound desc = could not find container \"58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c\": container with ID starting with 58b2db1a0b49bb458c1762a74e7aa6b022da8b20be2c38ec59b8e4a7b7ae7b4c not found: ID does not exist" Oct 30 14:45:11.591721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.591641 5824 scope.go:122] "RemoveContainer" containerID="22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c" Oct 30 14:45:11.601853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.601810 5824 scope.go:122] "RemoveContainer" containerID="22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c" Oct 30 14:45:11.602115 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:11.602097 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c\": container with ID starting with 22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c not found: ID does not exist" containerID="22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c" Oct 30 14:45:11.602213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.602123 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c"} err="failed to get container status \"22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c\": rpc error: code = NotFound desc = could not find container \"22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c\": container with ID starting with 22c04d995c54e420a57920270770c2bd9f3cf2e34070815f866c75a3d2cbb52c not found: ID does not exist" Oct 30 14:45:11.632121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.632086 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqxx\" (UniqueName: \"kubernetes.io/projected/592a7ead-f353-4b52-9fb2-f2bf41af4119-kube-api-access-xsqxx\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:11.638266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.638223 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" podStartSLOduration=1.638207914 podStartE2EDuration="1.638207914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:45:11.634605077 +0000 UTC m=+3341.435110519" watchObservedRunningTime="2026-10-30 14:45:11.638207914 +0000 UTC m=+3341.438713358" Oct 30 14:45:11.693773 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.693701 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf"] Oct 30 14:45:11.698887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.698862 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf"] Oct 30 14:45:11.737373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.737339 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz"] Oct 30 14:45:11.763211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.756309 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz"] Oct 30 14:45:11.915264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.915231 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:45:11.915454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:11.915281 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 14:45:12.322982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:12.322946 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b25e4f7-687c-429f-9395-951ebe00e027" path="/var/lib/kubelet/pods/4b25e4f7-687c-429f-9395-951ebe00e027/volumes" Oct 30 14:45:12.323497 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:12.323480 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592a7ead-f353-4b52-9fb2-f2bf41af4119" path="/var/lib/kubelet/pods/592a7ead-f353-4b52-9fb2-f2bf41af4119/volumes" Oct 30 14:45:12.378292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:12.378246 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-ca229c-746c6c44f5-6j4pz" podUID="4b25e4f7-687c-429f-9395-951ebe00e027" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.52:50052\" within 1s: context deadline exceeded" Oct 30 14:45:12.378476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:12.378246 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-ca229c-5c5c6f9fc6-rvdqf" podUID="592a7ead-f353-4b52-9fb2-f2bf41af4119" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.53:50052\" within 1s: context deadline exceeded" Oct 30 14:45:12.378476 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:12.378352 5824 logging.go:55] [core] [Channel #15371 SubChannel #15372]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.53:50052", ServerName: "10.134.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.53:50052: operation was canceled" Oct 30 14:45:12.378476 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:12.378358 5824 logging.go:55] [core] [Channel #15373 SubChannel #15374]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.52:50052", ServerName: "10.134.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.52:50052: operation was canceled" Oct 30 14:45:12.576657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:12.576622 5824 generic.go:356] "Generic (PLEG): container finished" podID="c51621a4-9cc5-4fb6-be52-903bee548a74" containerID="6f68355cd97ff88c056cb12c45f49ef90c8eee8ea12cfc6234ae0af8ae026c51" exitCode=0 Oct 30 14:45:12.577130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:12.576701 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" event={"ID":"c51621a4-9cc5-4fb6-be52-903bee548a74","Type":"ContainerDied","Data":"6f68355cd97ff88c056cb12c45f49ef90c8eee8ea12cfc6234ae0af8ae026c51"} Oct 30 14:45:12.915665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:12.915569 5824 patch_prober.go:29] interesting pod/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 14:45:12.915665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:12.915625 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 14:45:13.752659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.752635 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:13.886809 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.886734 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-xks-ca\") pod \"c51621a4-9cc5-4fb6-be52-903bee548a74\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " Oct 30 14:45:13.886809 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.886770 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51621a4-9cc5-4fb6-be52-903bee548a74-kube-api-access-jzvpq\" (UniqueName: \"kubernetes.io/projected/c51621a4-9cc5-4fb6-be52-903bee548a74-kube-api-access-jzvpq\") pod \"c51621a4-9cc5-4fb6-be52-903bee548a74\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " Oct 30 14:45:13.886997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.886894 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-openshift-ca\") pod \"c51621a4-9cc5-4fb6-be52-903bee548a74\" (UID: \"c51621a4-9cc5-4fb6-be52-903bee548a74\") " Oct 30 14:45:13.887301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.887277 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-openshift-ca" pod "c51621a4-9cc5-4fb6-be52-903bee548a74" (UID: "c51621a4-9cc5-4fb6-be52-903bee548a74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:13.887650 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.887631 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:13.888948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.888922 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-xks-ca" pod "c51621a4-9cc5-4fb6-be52-903bee548a74" (UID: "c51621a4-9cc5-4fb6-be52-903bee548a74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:45:13.889052 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.888951 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51621a4-9cc5-4fb6-be52-903bee548a74-kube-api-access-jzvpq" pod "c51621a4-9cc5-4fb6-be52-903bee548a74" (UID: "c51621a4-9cc5-4fb6-be52-903bee548a74"). InnerVolumeSpecName "kube-api-access-jzvpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:13.920531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.920507 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:13.920876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.920850 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:13.921537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.921517 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:45:13.989490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.989463 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51621a4-9cc5-4fb6-be52-903bee548a74-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:13.989643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:13.989561 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvpq\" (UniqueName: \"kubernetes.io/projected/c51621a4-9cc5-4fb6-be52-903bee548a74-kube-api-access-jzvpq\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:14.594950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:14.594843 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" Oct 30 14:45:14.595131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:14.594848 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5" event={"ID":"c51621a4-9cc5-4fb6-be52-903bee548a74","Type":"ContainerDied","Data":"1fb631d9e5a8a0fd25bbe1ff48f4a29141b43fc06517caa9309a9d9b9b01ab3d"} Oct 30 14:45:14.595131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:14.594992 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fb631d9e5a8a0fd25bbe1ff48f4a29141b43fc06517caa9309a9d9b9b01ab3d" Oct 30 14:45:15.309480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.309443 5824 scope.go:122] "RemoveContainer" containerID="f5ff391bc4f2d3d7c3f8108166ce1e610de8b57f3f035f3d6bbdf69d04701d65" Oct 30 14:45:15.438325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.438266 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" containerID="cri-o://82ffaca3073ffbb1db9df06893dc7730931903a22e008804314169602956a3ce" gracePeriod=45 Oct 30 14:45:15.604586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.604506 5824 generic.go:356] "Generic (PLEG): container finished" podID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerID="82ffaca3073ffbb1db9df06893dc7730931903a22e008804314169602956a3ce" exitCode=0 Oct 30 14:45:15.604750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.604582 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" event={"ID":"57e55e11-99aa-4bd3-8bb8-3f47a0273f43","Type":"ContainerDied","Data":"82ffaca3073ffbb1db9df06893dc7730931903a22e008804314169602956a3ce"} Oct 30 14:45:15.734577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.734554 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:45:15.818089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.818057 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-dshm\") pod \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " Oct 30 14:45:15.818320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.818114 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-home\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-home\") pod \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " Oct 30 14:45:15.818320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.818180 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-kube-api-access-v4kj6\" (UniqueName: \"kubernetes.io/projected/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-kube-api-access-v4kj6\") pod \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " Oct 30 14:45:15.818320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.818318 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-model-cache\") pod \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " Oct 30 14:45:15.818503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.818487 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tls-certs\" (UniqueName: \"kubernetes.io/secret/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tls-certs\") pod \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " Oct 30 14:45:15.818565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.818529 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tmp-dir\") pod \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\" (UID: \"57e55e11-99aa-4bd3-8bb8-3f47a0273f43\") " Oct 30 14:45:15.819025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.818975 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-home" pod "57e55e11-99aa-4bd3-8bb8-3f47a0273f43" (UID: "57e55e11-99aa-4bd3-8bb8-3f47a0273f43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.819153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.819135 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.819468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.819443 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tmp-dir" pod "57e55e11-99aa-4bd3-8bb8-3f47a0273f43" (UID: "57e55e11-99aa-4bd3-8bb8-3f47a0273f43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.819566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.819460 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-model-cache" pod "57e55e11-99aa-4bd3-8bb8-3f47a0273f43" (UID: "57e55e11-99aa-4bd3-8bb8-3f47a0273f43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.821337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.821316 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tls-certs" pod "57e55e11-99aa-4bd3-8bb8-3f47a0273f43" (UID: "57e55e11-99aa-4bd3-8bb8-3f47a0273f43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:45:15.821843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.821820 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-kube-api-access-v4kj6" pod "57e55e11-99aa-4bd3-8bb8-3f47a0273f43" (UID: "57e55e11-99aa-4bd3-8bb8-3f47a0273f43"). InnerVolumeSpecName "kube-api-access-v4kj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:15.821940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.821851 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-dshm" pod "57e55e11-99aa-4bd3-8bb8-3f47a0273f43" (UID: "57e55e11-99aa-4bd3-8bb8-3f47a0273f43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:45:15.920905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.920537 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.920905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.920569 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4kj6\" (UniqueName: \"kubernetes.io/projected/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-kube-api-access-v4kj6\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.920905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.920585 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.920905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.920598 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:15.920905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:15.920610 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e55e11-99aa-4bd3-8bb8-3f47a0273f43-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:16.309713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.309683 5824 scope.go:122] "RemoveContainer" containerID="b63c93c8f2a05f66f022d65fe2e1f31ab81775953c0a1429380ee5eeeeb6e382" Oct 30 14:45:16.310152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.309948 5824 scope.go:122] "RemoveContainer" containerID="cb2382b8661bc15794cd50d314975634423ebc025821ab3ff8756ad00faa2a48" Oct 30 14:45:16.310152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.310083 5824 scope.go:122] "RemoveContainer" containerID="3e9f49b254e2bbb48fc83cc0ef5e80603418bddd61132e0cff0938643f5e8654" Oct 30 14:45:16.614568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.614537 5824 generic.go:356] "Generic (PLEG): container finished" podID="6770fcff-79f8-4f91-a951-308c8ed76836" containerID="468c74c1bec053cb5eebc363e8b2669b5c786cc903af9041362581e14a0bc22c" exitCode=60 Oct 30 14:45:16.614715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.614614 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" event={"ID":"6770fcff-79f8-4f91-a951-308c8ed76836","Type":"ContainerDied","Data":"468c74c1bec053cb5eebc363e8b2669b5c786cc903af9041362581e14a0bc22c"} Oct 30 14:45:16.614715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.614683 5824 scope.go:122] "RemoveContainer" containerID="f5ff391bc4f2d3d7c3f8108166ce1e610de8b57f3f035f3d6bbdf69d04701d65" Oct 30 14:45:16.615143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.615117 5824 scope.go:122] "RemoveContainer" containerID="468c74c1bec053cb5eebc363e8b2669b5c786cc903af9041362581e14a0bc22c" Oct 30 14:45:16.615531 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:16.615510 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx_odh-ai-gateway-infra(6770fcff-79f8-4f91-a951-308c8ed76836)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" Oct 30 14:45:16.616358 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.616340 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" Oct 30 14:45:16.616477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.616388 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl" event={"ID":"57e55e11-99aa-4bd3-8bb8-3f47a0273f43","Type":"ContainerDied","Data":"1c459557e99e2f4cef131e4145421c4fd445df36807aaf519da2567e7c3ec181"} Oct 30 14:45:16.629659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.629637 5824 scope.go:122] "RemoveContainer" containerID="82ffaca3073ffbb1db9df06893dc7730931903a22e008804314169602956a3ce" Oct 30 14:45:16.700739 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.700715 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl"] Oct 30 14:45:16.707522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:16.707502 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ca229c/distinct-w1-ca229c-kserve-6f95878d64-qj5fl"] Oct 30 14:45:17.309492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.309470 5824 scope.go:122] "RemoveContainer" containerID="677a8ffd252a1733ddf2ce5ff1def506fa62507842df75f3f033c27eb4adad5f" Oct 30 14:45:17.309652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.309617 5824 scope.go:122] "RemoveContainer" containerID="deaca9f075dc69511d5838df07cf985da7f0b838c3f9019e6bb834fc889d0a30" Oct 30 14:45:17.627635 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.627604 5824 generic.go:356] "Generic (PLEG): container finished" podID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerID="710701d5512cb1bf165c39680e6a23b314300ef9a0e23b7032aaa59963856290" exitCode=60 Oct 30 14:45:17.628098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.627675 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" event={"ID":"3069b2d7-58d3-4095-ae46-c2e8d355931a","Type":"ContainerDied","Data":"710701d5512cb1bf165c39680e6a23b314300ef9a0e23b7032aaa59963856290"} Oct 30 14:45:17.628098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.627742 5824 scope.go:122] "RemoveContainer" containerID="cb2382b8661bc15794cd50d314975634423ebc025821ab3ff8756ad00faa2a48" Oct 30 14:45:17.628348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.628183 5824 scope.go:122] "RemoveContainer" containerID="710701d5512cb1bf165c39680e6a23b314300ef9a0e23b7032aaa59963856290" Oct 30 14:45:17.628624 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:17.628599 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8_odh-ai-gateway-infra(3069b2d7-58d3-4095-ae46-c2e8d355931a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" Oct 30 14:45:17.630573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.630548 5824 generic.go:356] "Generic (PLEG): container finished" podID="110d3615-218b-49be-b6b6-99062913a1df" containerID="b0e6e4d762cb6dfff6a833cb3d83a234e363287e8bd1949cb76ee38cff153455" exitCode=60 Oct 30 14:45:17.630702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.630676 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" event={"ID":"110d3615-218b-49be-b6b6-99062913a1df","Type":"ContainerDied","Data":"b0e6e4d762cb6dfff6a833cb3d83a234e363287e8bd1949cb76ee38cff153455"} Oct 30 14:45:17.631099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.631035 5824 scope.go:122] "RemoveContainer" containerID="b0e6e4d762cb6dfff6a833cb3d83a234e363287e8bd1949cb76ee38cff153455" Oct 30 14:45:17.631391 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:17.631369 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf_odh-ai-gateway-infra(110d3615-218b-49be-b6b6-99062913a1df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" podUID="110d3615-218b-49be-b6b6-99062913a1df" Oct 30 14:45:17.636184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.636165 5824 generic.go:356] "Generic (PLEG): container finished" podID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerID="40a832363eaed3dc412e5f3c12b1756adb3a974654208978ba66df4555e51e9b" exitCode=60 Oct 30 14:45:17.636284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.636230 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" event={"ID":"5076f0fd-cf19-46e1-8cb5-ab933b424acb","Type":"ContainerDied","Data":"40a832363eaed3dc412e5f3c12b1756adb3a974654208978ba66df4555e51e9b"} Oct 30 14:45:17.636602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.636584 5824 scope.go:122] "RemoveContainer" containerID="40a832363eaed3dc412e5f3c12b1756adb3a974654208978ba66df4555e51e9b" Oct 30 14:45:17.636845 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:17.636825 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl_odh-ai-gateway-infra(5076f0fd-cf19-46e1-8cb5-ab933b424acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" Oct 30 14:45:17.644431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.644335 5824 scope.go:122] "RemoveContainer" containerID="3e9f49b254e2bbb48fc83cc0ef5e80603418bddd61132e0cff0938643f5e8654" Oct 30 14:45:17.673407 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.672630 5824 scope.go:122] "RemoveContainer" containerID="b63c93c8f2a05f66f022d65fe2e1f31ab81775953c0a1429380ee5eeeeb6e382" Oct 30 14:45:17.678382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.678352 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx"] Oct 30 14:45:17.882841 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:17.882819 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:18.041533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.041498 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr"] Oct 30 14:45:18.053628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.053591 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46035f76-mg7hr"] Oct 30 14:45:18.069697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.069661 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6770fcff-79f8-4f91-a951-308c8ed76836-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6770fcff-79f8-4f91-a951-308c8ed76836-maas-api-service-ca\") pod \"6770fcff-79f8-4f91-a951-308c8ed76836\" (UID: \"6770fcff-79f8-4f91-a951-308c8ed76836\") " Oct 30 14:45:18.069956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.069934 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6770fcff-79f8-4f91-a951-308c8ed76836-kube-api-access-pgd6p\" (UniqueName: \"kubernetes.io/projected/6770fcff-79f8-4f91-a951-308c8ed76836-kube-api-access-pgd6p\") pod \"6770fcff-79f8-4f91-a951-308c8ed76836\" (UID: \"6770fcff-79f8-4f91-a951-308c8ed76836\") " Oct 30 14:45:18.070711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.070676 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6770fcff-79f8-4f91-a951-308c8ed76836-maas-api-service-ca" pod "6770fcff-79f8-4f91-a951-308c8ed76836" (UID: "6770fcff-79f8-4f91-a951-308c8ed76836"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:18.070946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.070918 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6770fcff-79f8-4f91-a951-308c8ed76836-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:18.072680 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.072648 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6770fcff-79f8-4f91-a951-308c8ed76836-kube-api-access-pgd6p" pod "6770fcff-79f8-4f91-a951-308c8ed76836" (UID: "6770fcff-79f8-4f91-a951-308c8ed76836"). InnerVolumeSpecName "kube-api-access-pgd6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:18.172816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.172731 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgd6p\" (UniqueName: \"kubernetes.io/projected/6770fcff-79f8-4f91-a951-308c8ed76836-kube-api-access-pgd6p\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:18.310050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.310011 5824 scope.go:122] "RemoveContainer" containerID="4b6d91a53bc02e9edceb4c04ae0ce6a3990ce62a6b00c9186c68bd8b22a52d75" Oct 30 14:45:18.310244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.310126 5824 scope.go:122] "RemoveContainer" containerID="d3046c10fd09e90d8a908b67462cd2b20287128d0f0d3481f064816f991af878" Oct 30 14:45:18.310244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.310175 5824 scope.go:122] "RemoveContainer" containerID="f430ea1f938f6fd8c67ce7b20151270dda85a1f1ab8f095eebee516c050587d9" Oct 30 14:45:18.310360 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.310301 5824 scope.go:122] "RemoveContainer" containerID="1c0820911d18800dd3ec11e6986d989fd650c97f41a6ee688d1baa4b17c02453" Oct 30 14:45:18.321502 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.321476 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" path="/var/lib/kubelet/pods/57e55e11-99aa-4bd3-8bb8-3f47a0273f43/volumes" Oct 30 14:45:18.322542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.322520 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42" path="/var/lib/kubelet/pods/a92ec28c-f4b8-4bcb-9fd4-df80e87dfd42/volumes" Oct 30 14:45:18.653593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.653551 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" event={"ID":"3661e6e3-d3ff-4e1d-a171-cd8ab00132be","Type":"ContainerStarted","Data":"65e0731dffebf5854947a8067d7d5b3a831b15a4b496d8bca3f526c52a27dc90"} Oct 30 14:45:18.656920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.655857 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" event={"ID":"6770fcff-79f8-4f91-a951-308c8ed76836","Type":"ContainerDied","Data":"1355fa676f27d7900e0c1b3f08c653f712d2104acfc68ceac6610b51e107c14c"} Oct 30 14:45:18.656920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.655903 5824 scope.go:122] "RemoveContainer" containerID="468c74c1bec053cb5eebc363e8b2669b5c786cc903af9041362581e14a0bc22c" Oct 30 14:45:18.656920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.655988 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx" Oct 30 14:45:18.664296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.664271 5824 generic.go:356] "Generic (PLEG): container finished" podID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerID="9b9f20f76f8c2cc7aa3bbfe1f921572c965656e64e1bffa79580462dbb8ebebc" exitCode=60 Oct 30 14:45:18.664415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.664353 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" event={"ID":"feee7683-a6bb-47fb-b085-6a510fc1b993","Type":"ContainerDied","Data":"9b9f20f76f8c2cc7aa3bbfe1f921572c965656e64e1bffa79580462dbb8ebebc"} Oct 30 14:45:18.664820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.664805 5824 scope.go:122] "RemoveContainer" containerID="9b9f20f76f8c2cc7aa3bbfe1f921572c965656e64e1bffa79580462dbb8ebebc" Oct 30 14:45:18.665076 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:18.665059 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952_odh-ai-gateway-infra(feee7683-a6bb-47fb-b085-6a510fc1b993)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" Oct 30 14:45:18.680277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.680255 5824 scope.go:122] "RemoveContainer" containerID="677a8ffd252a1733ddf2ce5ff1def506fa62507842df75f3f033c27eb4adad5f" Oct 30 14:45:18.686455 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.686435 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8"] Oct 30 14:45:18.731689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.731658 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl"] Oct 30 14:45:18.793793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.793153 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf"] Oct 30 14:45:18.804602 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.804572 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx"] Oct 30 14:45:18.814751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.814727 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0e8b9-29889525-qslqx"] Oct 30 14:45:18.997737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:18.997700 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:19.071575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.071545 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:19.088918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.088888 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5076f0fd-cf19-46e1-8cb5-ab933b424acb-kube-api-access-2gbrt\" (UniqueName: \"kubernetes.io/projected/5076f0fd-cf19-46e1-8cb5-ab933b424acb-kube-api-access-2gbrt\") pod \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\" (UID: \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\") " Oct 30 14:45:19.089216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.089170 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3069b2d7-58d3-4095-ae46-c2e8d355931a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069b2d7-58d3-4095-ae46-c2e8d355931a-maas-api-service-ca\") pod \"3069b2d7-58d3-4095-ae46-c2e8d355931a\" (UID: \"3069b2d7-58d3-4095-ae46-c2e8d355931a\") " Oct 30 14:45:19.089441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.089423 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3069b2d7-58d3-4095-ae46-c2e8d355931a-kube-api-access-zhnm5\" (UniqueName: \"kubernetes.io/projected/3069b2d7-58d3-4095-ae46-c2e8d355931a-kube-api-access-zhnm5\") pod \"3069b2d7-58d3-4095-ae46-c2e8d355931a\" (UID: \"3069b2d7-58d3-4095-ae46-c2e8d355931a\") " Oct 30 14:45:19.089946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.089562 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3069b2d7-58d3-4095-ae46-c2e8d355931a-maas-api-service-ca" pod "3069b2d7-58d3-4095-ae46-c2e8d355931a" (UID: "3069b2d7-58d3-4095-ae46-c2e8d355931a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:19.090127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.090100 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5076f0fd-cf19-46e1-8cb5-ab933b424acb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5076f0fd-cf19-46e1-8cb5-ab933b424acb-maas-api-service-ca\") pod \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\" (UID: \"5076f0fd-cf19-46e1-8cb5-ab933b424acb\") " Oct 30 14:45:19.091160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.090696 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5076f0fd-cf19-46e1-8cb5-ab933b424acb-maas-api-service-ca" pod "5076f0fd-cf19-46e1-8cb5-ab933b424acb" (UID: "5076f0fd-cf19-46e1-8cb5-ab933b424acb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:19.091160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.091075 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5076f0fd-cf19-46e1-8cb5-ab933b424acb-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.091160 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.091095 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3069b2d7-58d3-4095-ae46-c2e8d355931a-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.091418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.091376 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5076f0fd-cf19-46e1-8cb5-ab933b424acb-kube-api-access-2gbrt" pod "5076f0fd-cf19-46e1-8cb5-ab933b424acb" (UID: "5076f0fd-cf19-46e1-8cb5-ab933b424acb"). InnerVolumeSpecName "kube-api-access-2gbrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:19.091911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.091888 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3069b2d7-58d3-4095-ae46-c2e8d355931a-kube-api-access-zhnm5" pod "3069b2d7-58d3-4095-ae46-c2e8d355931a" (UID: "3069b2d7-58d3-4095-ae46-c2e8d355931a"). InnerVolumeSpecName "kube-api-access-zhnm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:19.110588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.110568 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:19.194958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.194922 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110d3615-218b-49be-b6b6-99062913a1df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d3615-218b-49be-b6b6-99062913a1df-maas-api-service-ca\") pod \"110d3615-218b-49be-b6b6-99062913a1df\" (UID: \"110d3615-218b-49be-b6b6-99062913a1df\") " Oct 30 14:45:19.195411 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.195380 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110d3615-218b-49be-b6b6-99062913a1df-maas-api-service-ca" pod "110d3615-218b-49be-b6b6-99062913a1df" (UID: "110d3615-218b-49be-b6b6-99062913a1df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:19.195560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.195541 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110d3615-218b-49be-b6b6-99062913a1df-kube-api-access-mdwgg\" (UniqueName: \"kubernetes.io/projected/110d3615-218b-49be-b6b6-99062913a1df-kube-api-access-mdwgg\") pod \"110d3615-218b-49be-b6b6-99062913a1df\" (UID: \"110d3615-218b-49be-b6b6-99062913a1df\") " Oct 30 14:45:19.196512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.196404 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gbrt\" (UniqueName: \"kubernetes.io/projected/5076f0fd-cf19-46e1-8cb5-ab933b424acb-kube-api-access-2gbrt\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.196512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.196433 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhnm5\" (UniqueName: \"kubernetes.io/projected/3069b2d7-58d3-4095-ae46-c2e8d355931a-kube-api-access-zhnm5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.196685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.196568 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d3615-218b-49be-b6b6-99062913a1df-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.198969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.198904 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110d3615-218b-49be-b6b6-99062913a1df-kube-api-access-mdwgg" pod "110d3615-218b-49be-b6b6-99062913a1df" (UID: "110d3615-218b-49be-b6b6-99062913a1df"). InnerVolumeSpecName "kube-api-access-mdwgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:19.299807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.299772 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwgg\" (UniqueName: \"kubernetes.io/projected/110d3615-218b-49be-b6b6-99062913a1df-kube-api-access-mdwgg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:19.309327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.309300 5824 scope.go:122] "RemoveContainer" containerID="8eda928c71dd2a8ae9b8099c4eaacf867346920c22669efe0d07455d74316157" Oct 30 14:45:19.675857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.675833 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" Oct 30 14:45:19.676259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.675849 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl" event={"ID":"5076f0fd-cf19-46e1-8cb5-ab933b424acb","Type":"ContainerDied","Data":"8614fc4f6a792eaee88017cb44382f76655f031e215f66d2d99fd471e6e3af61"} Oct 30 14:45:19.676259 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.675928 5824 scope.go:122] "RemoveContainer" containerID="40a832363eaed3dc412e5f3c12b1756adb3a974654208978ba66df4555e51e9b" Oct 30 14:45:19.678296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.678272 5824 generic.go:356] "Generic (PLEG): container finished" podID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerID="c3b728fbfa2f0acd70afd728488f148f71c2e11d96326f065c3fab0f20bb50b0" exitCode=60 Oct 30 14:45:19.678406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.678351 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" event={"ID":"bf5c08e1-364b-456a-9888-a26fdd1055cb","Type":"ContainerDied","Data":"c3b728fbfa2f0acd70afd728488f148f71c2e11d96326f065c3fab0f20bb50b0"} Oct 30 14:45:19.678851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.678831 5824 scope.go:122] "RemoveContainer" containerID="c3b728fbfa2f0acd70afd728488f148f71c2e11d96326f065c3fab0f20bb50b0" Oct 30 14:45:19.679254 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:19.679227 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws_odh-ai-gateway-infra(bf5c08e1-364b-456a-9888-a26fdd1055cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" Oct 30 14:45:19.680045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.680027 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" Oct 30 14:45:19.680045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.680040 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8" event={"ID":"3069b2d7-58d3-4095-ae46-c2e8d355931a","Type":"ContainerDied","Data":"c3145e04e1226d9a4917e4c4913e16e48ee1075576d15378e04adf88f9fd37ce"} Oct 30 14:45:19.684382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.684360 5824 generic.go:356] "Generic (PLEG): container finished" podID="e566451a-09cf-4392-9006-23a6444553d9" containerID="3f7668b89d4863e74e1fcb5d476508bc91f7d1d13e8fb629dcf8c9da104a9e4b" exitCode=60 Oct 30 14:45:19.684474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.684426 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" event={"ID":"e566451a-09cf-4392-9006-23a6444553d9","Type":"ContainerDied","Data":"3f7668b89d4863e74e1fcb5d476508bc91f7d1d13e8fb629dcf8c9da104a9e4b"} Oct 30 14:45:19.684797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.684782 5824 scope.go:122] "RemoveContainer" containerID="3f7668b89d4863e74e1fcb5d476508bc91f7d1d13e8fb629dcf8c9da104a9e4b" Oct 30 14:45:19.685042 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:19.685024 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq_odh-ai-gateway-infra(e566451a-09cf-4392-9006-23a6444553d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" podUID="e566451a-09cf-4392-9006-23a6444553d9" Oct 30 14:45:19.685726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.685713 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" Oct 30 14:45:19.685812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.685740 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf" event={"ID":"110d3615-218b-49be-b6b6-99062913a1df","Type":"ContainerDied","Data":"64fd98e87ecba83afb1b4ca0250a9c03e4a7f536805a20fdd908643b71ffc2e4"} Oct 30 14:45:19.687729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.687709 5824 generic.go:356] "Generic (PLEG): container finished" podID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerID="6db69807c2929cbd673a62eb397757de55cf55d6eb55e41125057ea0e434ed3a" exitCode=60 Oct 30 14:45:19.687814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.687773 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" event={"ID":"a0041a35-bbe1-40b1-ac40-f28760c6ec17","Type":"ContainerDied","Data":"6db69807c2929cbd673a62eb397757de55cf55d6eb55e41125057ea0e434ed3a"} Oct 30 14:45:19.688064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.688050 5824 scope.go:122] "RemoveContainer" containerID="6db69807c2929cbd673a62eb397757de55cf55d6eb55e41125057ea0e434ed3a" Oct 30 14:45:19.688339 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:19.688316 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w_odh-ai-gateway-infra(a0041a35-bbe1-40b1-ac40-f28760c6ec17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" Oct 30 14:45:19.689762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.689742 5824 generic.go:356] "Generic (PLEG): container finished" podID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerID="65e0731dffebf5854947a8067d7d5b3a831b15a4b496d8bca3f526c52a27dc90" exitCode=60 Oct 30 14:45:19.689846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.689808 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" event={"ID":"3661e6e3-d3ff-4e1d-a171-cd8ab00132be","Type":"ContainerDied","Data":"65e0731dffebf5854947a8067d7d5b3a831b15a4b496d8bca3f526c52a27dc90"} Oct 30 14:45:19.690088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.690071 5824 scope.go:122] "RemoveContainer" containerID="65e0731dffebf5854947a8067d7d5b3a831b15a4b496d8bca3f526c52a27dc90" Oct 30 14:45:19.690342 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:19.690323 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb_odh-ai-gateway-infra(3661e6e3-d3ff-4e1d-a171-cd8ab00132be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" Oct 30 14:45:19.691829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.691808 5824 generic.go:356] "Generic (PLEG): container finished" podID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerID="b150cb3ac23b17b926def0b3bd17be56c41daabf0a05250525a29c1db030dffb" exitCode=60 Oct 30 14:45:19.691938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.691874 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" event={"ID":"e6cc8620-529c-47e4-ad22-0d4f8e998bc0","Type":"ContainerDied","Data":"b150cb3ac23b17b926def0b3bd17be56c41daabf0a05250525a29c1db030dffb"} Oct 30 14:45:19.692243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.692225 5824 scope.go:122] "RemoveContainer" containerID="b150cb3ac23b17b926def0b3bd17be56c41daabf0a05250525a29c1db030dffb" Oct 30 14:45:19.692482 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:19.692462 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws_odh-ai-gateway-infra(e6cc8620-529c-47e4-ad22-0d4f8e998bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" Oct 30 14:45:19.715833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.715811 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb"] Oct 30 14:45:19.816055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.816024 5824 scope.go:122] "RemoveContainer" containerID="1c0820911d18800dd3ec11e6986d989fd650c97f41a6ee688d1baa4b17c02453" Oct 30 14:45:19.836656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.836634 5824 scope.go:122] "RemoveContainer" containerID="710701d5512cb1bf165c39680e6a23b314300ef9a0e23b7032aaa59963856290" Oct 30 14:45:19.850420 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.850393 5824 scope.go:122] "RemoveContainer" containerID="4b6d91a53bc02e9edceb4c04ae0ce6a3990ce62a6b00c9186c68bd8b22a52d75" Oct 30 14:45:19.875014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.874992 5824 scope.go:122] "RemoveContainer" containerID="b0e6e4d762cb6dfff6a833cb3d83a234e363287e8bd1949cb76ee38cff153455" Oct 30 14:45:19.881210 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.881150 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952"] Oct 30 14:45:19.892643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.892622 5824 scope.go:122] "RemoveContainer" containerID="d3046c10fd09e90d8a908b67462cd2b20287128d0f0d3481f064816f991af878" Oct 30 14:45:19.909371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.909349 5824 scope.go:122] "RemoveContainer" containerID="deaca9f075dc69511d5838df07cf985da7f0b838c3f9019e6bb834fc889d0a30" Oct 30 14:45:19.930761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.930732 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6"] Oct 30 14:45:19.934578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:19.934038 5824 scope.go:122] "RemoveContainer" containerID="f430ea1f938f6fd8c67ce7b20151270dda85a1f1ab8f095eebee516c050587d9" Oct 30 14:45:20.032626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.032593 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf"] Oct 30 14:45:20.042679 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.042644 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f1f206-29889525-7l6qf"] Oct 30 14:45:20.082522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.082468 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:20.114282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.114255 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feee7683-a6bb-47fb-b085-6a510fc1b993-kube-api-access-6mphs\" (UniqueName: \"kubernetes.io/projected/feee7683-a6bb-47fb-b085-6a510fc1b993-kube-api-access-6mphs\") pod \"feee7683-a6bb-47fb-b085-6a510fc1b993\" (UID: \"feee7683-a6bb-47fb-b085-6a510fc1b993\") " Oct 30 14:45:20.114477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.114463 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feee7683-a6bb-47fb-b085-6a510fc1b993-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feee7683-a6bb-47fb-b085-6a510fc1b993-maas-api-service-ca\") pod \"feee7683-a6bb-47fb-b085-6a510fc1b993\" (UID: \"feee7683-a6bb-47fb-b085-6a510fc1b993\") " Oct 30 14:45:20.114807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.114783 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feee7683-a6bb-47fb-b085-6a510fc1b993-maas-api-service-ca" pod "feee7683-a6bb-47fb-b085-6a510fc1b993" (UID: "feee7683-a6bb-47fb-b085-6a510fc1b993"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:20.114986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.114968 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feee7683-a6bb-47fb-b085-6a510fc1b993-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:20.116347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.116328 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feee7683-a6bb-47fb-b085-6a510fc1b993-kube-api-access-6mphs" pod "feee7683-a6bb-47fb-b085-6a510fc1b993" (UID: "feee7683-a6bb-47fb-b085-6a510fc1b993"). InnerVolumeSpecName "kube-api-access-6mphs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:20.172671 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.172630 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8"] Oct 30 14:45:20.178389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.178359 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1854b-29889525-zkvd8"] Oct 30 14:45:20.209090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.209052 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl"] Oct 30 14:45:20.215216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.215173 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-da78c4-29889525-l8jpl"] Oct 30 14:45:20.216167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.216140 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mphs\" (UniqueName: \"kubernetes.io/projected/feee7683-a6bb-47fb-b085-6a510fc1b993-kube-api-access-6mphs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:20.318320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.318287 5824 scope.go:122] "RemoveContainer" containerID="0e352f359fb1157a8c798c9be3651191c60a8e789033a65f1e2541d986155912" Oct 30 14:45:20.323247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.323176 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110d3615-218b-49be-b6b6-99062913a1df" path="/var/lib/kubelet/pods/110d3615-218b-49be-b6b6-99062913a1df/volumes" Oct 30 14:45:20.323876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.323855 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" path="/var/lib/kubelet/pods/3069b2d7-58d3-4095-ae46-c2e8d355931a/volumes" Oct 30 14:45:20.324505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.324484 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" path="/var/lib/kubelet/pods/5076f0fd-cf19-46e1-8cb5-ab933b424acb/volumes" Oct 30 14:45:20.325484 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.325460 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" path="/var/lib/kubelet/pods/6770fcff-79f8-4f91-a951-308c8ed76836/volumes" Oct 30 14:45:20.728296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.728097 5824 generic.go:356] "Generic (PLEG): container finished" podID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerID="3d988e4eb179e17b55811ad130972b9fa1f8436f4ae6850f4f91777647c4520d" exitCode=60 Oct 30 14:45:20.729806 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.728177 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" event={"ID":"7acc1072-69bf-4a93-8428-90ba63f5dea8","Type":"ContainerDied","Data":"3d988e4eb179e17b55811ad130972b9fa1f8436f4ae6850f4f91777647c4520d"} Oct 30 14:45:20.729806 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.728861 5824 scope.go:122] "RemoveContainer" containerID="8eda928c71dd2a8ae9b8099c4eaacf867346920c22669efe0d07455d74316157" Oct 30 14:45:20.729806 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.729358 5824 scope.go:122] "RemoveContainer" containerID="3d988e4eb179e17b55811ad130972b9fa1f8436f4ae6850f4f91777647c4520d" Oct 30 14:45:20.729806 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:20.729668 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh_odh-ai-gateway-infra(7acc1072-69bf-4a93-8428-90ba63f5dea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" Oct 30 14:45:20.770919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.770892 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws"] Oct 30 14:45:20.776024 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.775993 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" event={"ID":"feee7683-a6bb-47fb-b085-6a510fc1b993","Type":"ContainerDied","Data":"34574efb78fe558798a410fcb553bf610d95f881f73dc3f01460f746cbc4bf99"} Oct 30 14:45:20.776216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.776057 5824 scope.go:122] "RemoveContainer" containerID="9b9f20f76f8c2cc7aa3bbfe1f921572c965656e64e1bffa79580462dbb8ebebc" Oct 30 14:45:20.776216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.776165 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952" Oct 30 14:45:20.879221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.877655 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws"] Oct 30 14:45:20.941536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.941513 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq"] Oct 30 14:45:20.963305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.963265 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952"] Oct 30 14:45:20.967962 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:20.967909 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a59ecb-29889525-29952"] Oct 30 14:45:21.014741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.014664 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:21.018813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.018741 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w"] Oct 30 14:45:21.089789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.089760 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:21.132409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.132382 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-maas-api-service-ca\") pod \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\" (UID: \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\") " Oct 30 14:45:21.132481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.132451 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-kube-api-access-zgqmv\" (UniqueName: \"kubernetes.io/projected/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-kube-api-access-zgqmv\") pod \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\" (UID: \"3661e6e3-d3ff-4e1d-a171-cd8ab00132be\") " Oct 30 14:45:21.132524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.132508 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-maas-api-service-ca\") pod \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\" (UID: \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\") " Oct 30 14:45:21.132607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.132587 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-kube-api-access-fj5pm\" (UniqueName: \"kubernetes.io/projected/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-kube-api-access-fj5pm\") pod \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\" (UID: \"e6cc8620-529c-47e4-ad22-0d4f8e998bc0\") " Oct 30 14:45:21.134576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.133923 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-maas-api-service-ca" pod "3661e6e3-d3ff-4e1d-a171-cd8ab00132be" (UID: "3661e6e3-d3ff-4e1d-a171-cd8ab00132be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:21.134736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.134303 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-maas-api-service-ca" pod "e6cc8620-529c-47e4-ad22-0d4f8e998bc0" (UID: "e6cc8620-529c-47e4-ad22-0d4f8e998bc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:21.135442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.135418 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.135518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.135448 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.135806 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.135790 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-kube-api-access-zgqmv" pod "3661e6e3-d3ff-4e1d-a171-cd8ab00132be" (UID: "3661e6e3-d3ff-4e1d-a171-cd8ab00132be"). InnerVolumeSpecName "kube-api-access-zgqmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:21.138114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.138099 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-kube-api-access-fj5pm" pod "e6cc8620-529c-47e4-ad22-0d4f8e998bc0" (UID: "e6cc8620-529c-47e4-ad22-0d4f8e998bc0"). InnerVolumeSpecName "kube-api-access-fj5pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:21.146979 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.146963 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:21.197213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.197161 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:21.235887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.235864 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5c08e1-364b-456a-9888-a26fdd1055cb-kube-api-access-j4hxb\" (UniqueName: \"kubernetes.io/projected/bf5c08e1-364b-456a-9888-a26fdd1055cb-kube-api-access-j4hxb\") pod \"bf5c08e1-364b-456a-9888-a26fdd1055cb\" (UID: \"bf5c08e1-364b-456a-9888-a26fdd1055cb\") " Oct 30 14:45:21.236021 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.235909 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5c08e1-364b-456a-9888-a26fdd1055cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5c08e1-364b-456a-9888-a26fdd1055cb-maas-api-service-ca\") pod \"bf5c08e1-364b-456a-9888-a26fdd1055cb\" (UID: \"bf5c08e1-364b-456a-9888-a26fdd1055cb\") " Oct 30 14:45:21.236299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.236275 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e566451a-09cf-4392-9006-23a6444553d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e566451a-09cf-4392-9006-23a6444553d9-maas-api-service-ca\") pod \"e566451a-09cf-4392-9006-23a6444553d9\" (UID: \"e566451a-09cf-4392-9006-23a6444553d9\") " Oct 30 14:45:21.236531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.236505 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e566451a-09cf-4392-9006-23a6444553d9-kube-api-access-sf4k5\" (UniqueName: \"kubernetes.io/projected/e566451a-09cf-4392-9006-23a6444553d9-kube-api-access-sf4k5\") pod \"e566451a-09cf-4392-9006-23a6444553d9\" (UID: \"e566451a-09cf-4392-9006-23a6444553d9\") " Oct 30 14:45:21.236687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.236666 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e566451a-09cf-4392-9006-23a6444553d9-maas-api-service-ca" pod "e566451a-09cf-4392-9006-23a6444553d9" (UID: "e566451a-09cf-4392-9006-23a6444553d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:21.237007 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.236979 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5c08e1-364b-456a-9888-a26fdd1055cb-maas-api-service-ca" pod "bf5c08e1-364b-456a-9888-a26fdd1055cb" (UID: "bf5c08e1-364b-456a-9888-a26fdd1055cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:21.237321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.237299 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e566451a-09cf-4392-9006-23a6444553d9-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.237457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.237324 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqmv\" (UniqueName: \"kubernetes.io/projected/3661e6e3-d3ff-4e1d-a171-cd8ab00132be-kube-api-access-zgqmv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.237457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.237339 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj5pm\" (UniqueName: \"kubernetes.io/projected/e6cc8620-529c-47e4-ad22-0d4f8e998bc0-kube-api-access-fj5pm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.237457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.237353 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5c08e1-364b-456a-9888-a26fdd1055cb-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.238332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.238304 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5c08e1-364b-456a-9888-a26fdd1055cb-kube-api-access-j4hxb" pod "bf5c08e1-364b-456a-9888-a26fdd1055cb" (UID: "bf5c08e1-364b-456a-9888-a26fdd1055cb"). InnerVolumeSpecName "kube-api-access-j4hxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:21.239328 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.239304 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e566451a-09cf-4392-9006-23a6444553d9-kube-api-access-sf4k5" pod "e566451a-09cf-4392-9006-23a6444553d9" (UID: "e566451a-09cf-4392-9006-23a6444553d9"). InnerVolumeSpecName "kube-api-access-sf4k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:21.246730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.246683 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:21.338348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.338311 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0041a35-bbe1-40b1-ac40-f28760c6ec17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0041a35-bbe1-40b1-ac40-f28760c6ec17-maas-api-service-ca\") pod \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\" (UID: \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\") " Oct 30 14:45:21.338681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.338659 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0041a35-bbe1-40b1-ac40-f28760c6ec17-kube-api-access-gf4wf\" (UniqueName: \"kubernetes.io/projected/a0041a35-bbe1-40b1-ac40-f28760c6ec17-kube-api-access-gf4wf\") pod \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\" (UID: \"a0041a35-bbe1-40b1-ac40-f28760c6ec17\") " Oct 30 14:45:21.338810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.338693 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0041a35-bbe1-40b1-ac40-f28760c6ec17-maas-api-service-ca" pod "a0041a35-bbe1-40b1-ac40-f28760c6ec17" (UID: "a0041a35-bbe1-40b1-ac40-f28760c6ec17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:21.339427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.339402 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf4k5\" (UniqueName: \"kubernetes.io/projected/e566451a-09cf-4392-9006-23a6444553d9-kube-api-access-sf4k5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.339543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.339433 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4hxb\" (UniqueName: \"kubernetes.io/projected/bf5c08e1-364b-456a-9888-a26fdd1055cb-kube-api-access-j4hxb\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.339543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.339448 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0041a35-bbe1-40b1-ac40-f28760c6ec17-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.345132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.344635 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0041a35-bbe1-40b1-ac40-f28760c6ec17-kube-api-access-gf4wf" pod "a0041a35-bbe1-40b1-ac40-f28760c6ec17" (UID: "a0041a35-bbe1-40b1-ac40-f28760c6ec17"). InnerVolumeSpecName "kube-api-access-gf4wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:21.441345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.441310 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf4wf\" (UniqueName: \"kubernetes.io/projected/a0041a35-bbe1-40b1-ac40-f28760c6ec17-kube-api-access-gf4wf\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:21.792517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.792488 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" Oct 30 14:45:21.792517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.792501 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws" event={"ID":"bf5c08e1-364b-456a-9888-a26fdd1055cb","Type":"ContainerDied","Data":"6e466bf5ca736203e31e35e1a766192f8e1ffb6e4a91385de5999e8b9ae7050e"} Oct 30 14:45:21.793138 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.792642 5824 scope.go:122] "RemoveContainer" containerID="c3b728fbfa2f0acd70afd728488f148f71c2e11d96326f065c3fab0f20bb50b0" Oct 30 14:45:21.794877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.794852 5824 generic.go:356] "Generic (PLEG): container finished" podID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerID="9b32283909bd0d8c83460b185b008d85fac1d9de35068f65c53c2a7ad9b438eb" exitCode=60 Oct 30 14:45:21.795033 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.794947 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" event={"ID":"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e","Type":"ContainerDied","Data":"9b32283909bd0d8c83460b185b008d85fac1d9de35068f65c53c2a7ad9b438eb"} Oct 30 14:45:21.795425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.795403 5824 scope.go:122] "RemoveContainer" containerID="9b32283909bd0d8c83460b185b008d85fac1d9de35068f65c53c2a7ad9b438eb" Oct 30 14:45:21.795687 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:21.795667 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9_odh-ai-gateway-infra(6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" Oct 30 14:45:21.797431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.797396 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" event={"ID":"e566451a-09cf-4392-9006-23a6444553d9","Type":"ContainerDied","Data":"c17673dfc46e4c2bf9bbee593e95892c5f3dc9b4428ab031ff875ccf12f47289"} Oct 30 14:45:21.797541 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.797523 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq" Oct 30 14:45:21.804708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.804683 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" event={"ID":"a0041a35-bbe1-40b1-ac40-f28760c6ec17","Type":"ContainerDied","Data":"432cf5ad9406495afd060dd4d17c0a8bcf36385a0cd09304b7636d63b0ac52a6"} Oct 30 14:45:21.804802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.804790 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w" Oct 30 14:45:21.806720 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.806695 5824 scope.go:122] "RemoveContainer" containerID="0e352f359fb1157a8c798c9be3651191c60a8e789033a65f1e2541d986155912" Oct 30 14:45:21.808777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.808171 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" Oct 30 14:45:21.808898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.808874 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb" event={"ID":"3661e6e3-d3ff-4e1d-a171-cd8ab00132be","Type":"ContainerDied","Data":"92a402bfdc58f6bc4b6a95fe47b8fb310dd6f2735563d001af57efd3e03af3c0"} Oct 30 14:45:21.809392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.809349 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh"] Oct 30 14:45:21.810588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.810480 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" event={"ID":"e6cc8620-529c-47e4-ad22-0d4f8e998bc0","Type":"ContainerDied","Data":"981d5a3a83277346646314434166fe26194eef9ed2bd378c3583e200808a715e"} Oct 30 14:45:21.810588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.810508 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws" Oct 30 14:45:21.847335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.847311 5824 scope.go:122] "RemoveContainer" containerID="3f7668b89d4863e74e1fcb5d476508bc91f7d1d13e8fb629dcf8c9da104a9e4b" Oct 30 14:45:21.864876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.864771 5824 scope.go:122] "RemoveContainer" containerID="6db69807c2929cbd673a62eb397757de55cf55d6eb55e41125057ea0e434ed3a" Oct 30 14:45:21.878332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.878311 5824 scope.go:122] "RemoveContainer" containerID="65e0731dffebf5854947a8067d7d5b3a831b15a4b496d8bca3f526c52a27dc90" Oct 30 14:45:21.890419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.890397 5824 scope.go:122] "RemoveContainer" containerID="b150cb3ac23b17b926def0b3bd17be56c41daabf0a05250525a29c1db030dffb" Oct 30 14:45:21.909326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.909293 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws"] Oct 30 14:45:21.916453 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:21.916112 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-39529e-29889525-rgpws"] Oct 30 14:45:22.004963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.004940 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:22.052327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.052261 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7acc1072-69bf-4a93-8428-90ba63f5dea8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acc1072-69bf-4a93-8428-90ba63f5dea8-maas-api-service-ca\") pod \"7acc1072-69bf-4a93-8428-90ba63f5dea8\" (UID: \"7acc1072-69bf-4a93-8428-90ba63f5dea8\") " Oct 30 14:45:22.052455 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.052374 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acc1072-69bf-4a93-8428-90ba63f5dea8-kube-api-access-jbv47\" (UniqueName: \"kubernetes.io/projected/7acc1072-69bf-4a93-8428-90ba63f5dea8-kube-api-access-jbv47\") pod \"7acc1072-69bf-4a93-8428-90ba63f5dea8\" (UID: \"7acc1072-69bf-4a93-8428-90ba63f5dea8\") " Oct 30 14:45:22.052638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.052612 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7acc1072-69bf-4a93-8428-90ba63f5dea8-maas-api-service-ca" pod "7acc1072-69bf-4a93-8428-90ba63f5dea8" (UID: "7acc1072-69bf-4a93-8428-90ba63f5dea8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:22.052918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.052894 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7acc1072-69bf-4a93-8428-90ba63f5dea8-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:22.054359 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.054336 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acc1072-69bf-4a93-8428-90ba63f5dea8-kube-api-access-jbv47" pod "7acc1072-69bf-4a93-8428-90ba63f5dea8" (UID: "7acc1072-69bf-4a93-8428-90ba63f5dea8"). InnerVolumeSpecName "kube-api-access-jbv47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:22.088200 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.088171 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w"] Oct 30 14:45:22.095531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.095510 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2df0ab-29889525-khj5w"] Oct 30 14:45:22.134731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.134709 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws"] Oct 30 14:45:22.139837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.139812 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-396d70-29889525-gpbws"] Oct 30 14:45:22.153877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.153854 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbv47\" (UniqueName: \"kubernetes.io/projected/7acc1072-69bf-4a93-8428-90ba63f5dea8-kube-api-access-jbv47\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:22.178623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.178599 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb"] Oct 30 14:45:22.184394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.184368 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9e3aa0-29889525-dxqrb"] Oct 30 14:45:22.224135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.224110 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq"] Oct 30 14:45:22.227281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.227261 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0efe43-29889525-htwnq"] Oct 30 14:45:22.320865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.320790 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" path="/var/lib/kubelet/pods/3661e6e3-d3ff-4e1d-a171-cd8ab00132be/volumes" Oct 30 14:45:22.321321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.321300 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" path="/var/lib/kubelet/pods/a0041a35-bbe1-40b1-ac40-f28760c6ec17/volumes" Oct 30 14:45:22.321756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.321740 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" path="/var/lib/kubelet/pods/bf5c08e1-364b-456a-9888-a26fdd1055cb/volumes" Oct 30 14:45:22.322530 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.322513 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e566451a-09cf-4392-9006-23a6444553d9" path="/var/lib/kubelet/pods/e566451a-09cf-4392-9006-23a6444553d9/volumes" Oct 30 14:45:22.322916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.322902 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" path="/var/lib/kubelet/pods/e6cc8620-529c-47e4-ad22-0d4f8e998bc0/volumes" Oct 30 14:45:22.323347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.323333 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" path="/var/lib/kubelet/pods/feee7683-a6bb-47fb-b085-6a510fc1b993/volumes" Oct 30 14:45:22.823029 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.822992 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" event={"ID":"7acc1072-69bf-4a93-8428-90ba63f5dea8","Type":"ContainerDied","Data":"4246b75c4b630bd5cbdec45711b6ef7adcc0f4fcf830b22ffb9217ea67fa22ed"} Oct 30 14:45:22.823575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.823072 5824 scope.go:122] "RemoveContainer" containerID="3d988e4eb179e17b55811ad130972b9fa1f8436f4ae6850f4f91777647c4520d" Oct 30 14:45:22.823575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.823001 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh" Oct 30 14:45:22.885392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.885370 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh"] Oct 30 14:45:22.890137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:22.890116 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-228454-29889525-6k6bh"] Oct 30 14:45:23.058252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.058219 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9"] Oct 30 14:45:23.222642 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.222611 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:23.267903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.267877 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-maas-api-service-ca\") pod \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\" (UID: \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\") " Oct 30 14:45:23.268178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.267919 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-kube-api-access-45v52\" (UniqueName: \"kubernetes.io/projected/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-kube-api-access-45v52\") pod \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\" (UID: \"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e\") " Oct 30 14:45:23.268639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.268613 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-maas-api-service-ca" pod "6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" (UID: "6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:23.279668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.275383 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-kube-api-access-45v52" pod "6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" (UID: "6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e"). InnerVolumeSpecName "kube-api-access-45v52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:23.370086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.370012 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:23.370086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.370041 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45v52\" (UniqueName: \"kubernetes.io/projected/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e-kube-api-access-45v52\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:23.836109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.836078 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" Oct 30 14:45:23.836109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.836104 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9" event={"ID":"6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e","Type":"ContainerDied","Data":"ffe78aedaf307bf68827963dd05a5889b02599ff8dbb5fb68cc4677faecb50a9"} Oct 30 14:45:23.836676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.836160 5824 scope.go:122] "RemoveContainer" containerID="9b32283909bd0d8c83460b185b008d85fac1d9de35068f65c53c2a7ad9b438eb" Oct 30 14:45:23.896382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.896357 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9"] Oct 30 14:45:23.909601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:23.909576 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4be240-29889525-vswt9"] Oct 30 14:45:24.319308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:24.319274 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" path="/var/lib/kubelet/pods/6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e/volumes" Oct 30 14:45:24.319717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:24.319701 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" path="/var/lib/kubelet/pods/7acc1072-69bf-4a93-8428-90ba63f5dea8/volumes" Oct 30 14:45:24.932399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:24.932351 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" podUID="6f0597f7-23b7-4be4-af5e-4082e092826b" containerName="payload-processing" containerID="cri-o://4a895b9d800bed2155bd0abbb496f38a39f2d30bbaa2bce9979a95f77803c120" gracePeriod=25 Oct 30 14:45:28.042148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:28.042111 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg"] Oct 30 14:45:28.055297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:28.055272 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547a94e-v6hzg"] Oct 30 14:45:28.319536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:28.319465 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9a443b-f8c2-4fa6-89fd-06d43f9e5428" path="/var/lib/kubelet/pods/db9a443b-f8c2-4fa6-89fd-06d43f9e5428/volumes" Oct 30 14:45:32.038686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:32.038651 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" podUID="6f0597f7-23b7-4be4-af5e-4082e092826b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:45:33.645066 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:45:33.645014 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:45:36.528565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.528530 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm"] Oct 30 14:45:36.529262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529246 5824 cpu_manager.go:414] "containerMap: removing container" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:36.529262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529263 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529273 5824 cpu_manager.go:414] "containerMap: removing container" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529278 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529291 5824 cpu_manager.go:414] "containerMap: removing container" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529296 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529303 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529308 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529319 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529323 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529330 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529334 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529340 5824 cpu_manager.go:414] "containerMap: removing container" podUID="592a7ead-f353-4b52-9fb2-f2bf41af4119" containerName="payload-processing" Oct 30 14:45:36.529345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529345 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592a7ead-f353-4b52-9fb2-f2bf41af4119" containerName="payload-processing" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529352 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529357 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529367 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529371 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529381 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529385 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529391 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529395 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529401 5824 cpu_manager.go:414] "containerMap: removing container" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529405 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529415 5824 cpu_manager.go:414] "containerMap: removing container" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529419 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529425 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529430 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529440 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529444 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529454 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529458 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529468 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529472 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529478 5824 cpu_manager.go:414] "containerMap: removing container" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529482 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529488 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529492 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529500 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4b25e4f7-687c-429f-9395-951ebe00e027" containerName="payload-pre-processing" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529504 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b25e4f7-687c-429f-9395-951ebe00e027" containerName="payload-pre-processing" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529511 5824 cpu_manager.go:414] "containerMap: removing container" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529515 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529520 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529524 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529531 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c51621a4-9cc5-4fb6-be52-903bee548a74" containerName="revoke-keys" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529536 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51621a4-9cc5-4fb6-be52-903bee548a74" containerName="revoke-keys" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529543 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529547 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529555 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529558 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529569 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529574 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529587 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529591 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529597 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529601 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529609 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529615 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529626 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529634 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529644 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529650 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529659 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529663 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529672 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:36.529711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529676 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529788 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529797 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529803 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529808 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529814 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529818 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529824 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529830 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529834 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529839 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529844 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529848 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529854 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529858 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529863 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529870 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529876 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="592a7ead-f353-4b52-9fb2-f2bf41af4119" containerName="payload-processing" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529880 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529886 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3069b2d7-58d3-4095-ae46-c2e8d355931a" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529892 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529896 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529903 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529909 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529913 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529919 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6770fcff-79f8-4f91-a951-308c8ed76836" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529925 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529930 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529936 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529941 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e55e11-99aa-4bd3-8bb8-3f47a0273f43" containerName="main" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529946 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e566451a-09cf-4392-9006-23a6444553d9" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529952 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b25e4f7-687c-429f-9395-951ebe00e027" containerName="payload-pre-processing" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529957 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529961 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529967 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51621a4-9cc5-4fb6-be52-903bee548a74" containerName="revoke-keys" Oct 30 14:45:36.531239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.529971 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:36.533687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.533671 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.552985 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.552955 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm"] Oct 30 14:45:36.560907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.560880 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:45:36.561926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.561905 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:45:36.565952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.565919 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.566113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.566092 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jpk\" (UniqueName: \"kubernetes.io/projected/6c955a5d-f936-4233-a91b-7843ae0e5809-kube-api-access-95jpk\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.566351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.566329 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.667739 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.667704 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.667925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.667812 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.667925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.667909 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jpk\" (UniqueName: \"kubernetes.io/projected/6c955a5d-f936-4233-a91b-7843ae0e5809-kube-api-access-95jpk\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.668404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.668385 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.670169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.670148 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.693716 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.693693 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jpk\" (UniqueName: \"kubernetes.io/projected/6c955a5d-f936-4233-a91b-7843ae0e5809-kube-api-access-95jpk\") pod \"maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:36.849226 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:36.849137 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:37.160179 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:37.160148 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c955a5d_f936_4233_a91b_7843ae0e5809.slice/crio-c1207e701eb0339ea4c10794bf1c1d13b2237a51b87d52e3a1f262b7f63f0594 WatchSource:0}: Error finding container c1207e701eb0339ea4c10794bf1c1d13b2237a51b87d52e3a1f262b7f63f0594: Status 404 returned error can't find the container with id c1207e701eb0339ea4c10794bf1c1d13b2237a51b87d52e3a1f262b7f63f0594 Oct 30 14:45:37.162061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:37.162031 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm"] Oct 30 14:45:37.991948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:37.991910 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" event={"ID":"6c955a5d-f936-4233-a91b-7843ae0e5809","Type":"ContainerStarted","Data":"d7133909b2db4b564983cfca1b4fe279e588ce86b39f5ec33e8ab11d0217ff4e"} Oct 30 14:45:37.992396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:37.992006 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" event={"ID":"6c955a5d-f936-4233-a91b-7843ae0e5809","Type":"ContainerStarted","Data":"c1207e701eb0339ea4c10794bf1c1d13b2237a51b87d52e3a1f262b7f63f0594"} Oct 30 14:45:39.021260 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:39.021181 5824 generic.go:356] "Generic (PLEG): container finished" podID="6c955a5d-f936-4233-a91b-7843ae0e5809" containerID="d7133909b2db4b564983cfca1b4fe279e588ce86b39f5ec33e8ab11d0217ff4e" exitCode=0 Oct 30 14:45:39.021830 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:39.021389 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" event={"ID":"6c955a5d-f936-4233-a91b-7843ae0e5809","Type":"ContainerDied","Data":"d7133909b2db4b564983cfca1b4fe279e588ce86b39f5ec33e8ab11d0217ff4e"} Oct 30 14:45:40.207235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.207208 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:40.336770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.336695 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c955a5d-f936-4233-a91b-7843ae0e5809-kube-api-access-95jpk\" (UniqueName: \"kubernetes.io/projected/6c955a5d-f936-4233-a91b-7843ae0e5809-kube-api-access-95jpk\") pod \"6c955a5d-f936-4233-a91b-7843ae0e5809\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " Oct 30 14:45:40.336944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.336829 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-xks-ca\") pod \"6c955a5d-f936-4233-a91b-7843ae0e5809\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " Oct 30 14:45:40.336944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.336896 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-openshift-ca\") pod \"6c955a5d-f936-4233-a91b-7843ae0e5809\" (UID: \"6c955a5d-f936-4233-a91b-7843ae0e5809\") " Oct 30 14:45:40.337342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.337314 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-openshift-ca" pod "6c955a5d-f936-4233-a91b-7843ae0e5809" (UID: "6c955a5d-f936-4233-a91b-7843ae0e5809"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:40.337498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.337483 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:40.338788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.338763 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-xks-ca" pod "6c955a5d-f936-4233-a91b-7843ae0e5809" (UID: "6c955a5d-f936-4233-a91b-7843ae0e5809"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:45:40.338870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.338797 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c955a5d-f936-4233-a91b-7843ae0e5809-kube-api-access-95jpk" pod "6c955a5d-f936-4233-a91b-7843ae0e5809" (UID: "6c955a5d-f936-4233-a91b-7843ae0e5809"). InnerVolumeSpecName "kube-api-access-95jpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:40.438891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.438862 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95jpk\" (UniqueName: \"kubernetes.io/projected/6c955a5d-f936-4233-a91b-7843ae0e5809-kube-api-access-95jpk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:40.438891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:40.438892 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c955a5d-f936-4233-a91b-7843ae0e5809-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:41.037100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:41.037062 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" Oct 30 14:45:41.037100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:41.037088 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm" event={"ID":"6c955a5d-f936-4233-a91b-7843ae0e5809","Type":"ContainerDied","Data":"c1207e701eb0339ea4c10794bf1c1d13b2237a51b87d52e3a1f262b7f63f0594"} Oct 30 14:45:41.037387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:41.037134 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1207e701eb0339ea4c10794bf1c1d13b2237a51b87d52e3a1f262b7f63f0594" Oct 30 14:45:42.038912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:42.038868 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" podUID="6f0597f7-23b7-4be4-af5e-4082e092826b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:45:45.070316 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.070284 5824 generic.go:356] "Generic (PLEG): container finished" podID="6f0597f7-23b7-4be4-af5e-4082e092826b" containerID="4a895b9d800bed2155bd0abbb496f38a39f2d30bbaa2bce9979a95f77803c120" exitCode=0 Oct 30 14:45:45.070649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.070360 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" event={"ID":"6f0597f7-23b7-4be4-af5e-4082e092826b","Type":"ContainerDied","Data":"4a895b9d800bed2155bd0abbb496f38a39f2d30bbaa2bce9979a95f77803c120"} Oct 30 14:45:45.213631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.213607 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:45:45.318988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.318947 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0597f7-23b7-4be4-af5e-4082e092826b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0597f7-23b7-4be4-af5e-4082e092826b-plugins-config-volume\") pod \"6f0597f7-23b7-4be4-af5e-4082e092826b\" (UID: \"6f0597f7-23b7-4be4-af5e-4082e092826b\") " Oct 30 14:45:45.319285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.319267 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0597f7-23b7-4be4-af5e-4082e092826b-kube-api-access-ssj54\" (UniqueName: \"kubernetes.io/projected/6f0597f7-23b7-4be4-af5e-4082e092826b-kube-api-access-ssj54\") pod \"6f0597f7-23b7-4be4-af5e-4082e092826b\" (UID: \"6f0597f7-23b7-4be4-af5e-4082e092826b\") " Oct 30 14:45:45.319553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.319396 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0597f7-23b7-4be4-af5e-4082e092826b-plugins-config-volume" pod "6f0597f7-23b7-4be4-af5e-4082e092826b" (UID: "6f0597f7-23b7-4be4-af5e-4082e092826b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:45:45.319862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.319842 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0597f7-23b7-4be4-af5e-4082e092826b-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:45.321692 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.321667 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0597f7-23b7-4be4-af5e-4082e092826b-kube-api-access-ssj54" pod "6f0597f7-23b7-4be4-af5e-4082e092826b" (UID: "6f0597f7-23b7-4be4-af5e-4082e092826b"). InnerVolumeSpecName "kube-api-access-ssj54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:45:45.420942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:45.420908 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssj54\" (UniqueName: \"kubernetes.io/projected/6f0597f7-23b7-4be4-af5e-4082e092826b-kube-api-access-ssj54\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:45:46.078913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:46.078883 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" Oct 30 14:45:46.079352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:46.078917 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6" event={"ID":"6f0597f7-23b7-4be4-af5e-4082e092826b","Type":"ContainerDied","Data":"4642363786edb09e35cdcd74b4ff07cd67a0f4287e03a6e7c19cfe0e85f59a10"} Oct 30 14:45:46.079352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:46.078983 5824 scope.go:122] "RemoveContainer" containerID="4a895b9d800bed2155bd0abbb496f38a39f2d30bbaa2bce9979a95f77803c120" Oct 30 14:45:46.140515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:46.140484 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6"] Oct 30 14:45:46.154733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:46.154700 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-1d6347-854b8c4d9d-zljd6"] Oct 30 14:45:46.318629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:46.318594 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0597f7-23b7-4be4-af5e-4082e092826b" path="/var/lib/kubelet/pods/6f0597f7-23b7-4be4-af5e-4082e092826b/volumes" Oct 30 14:45:57.448052 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448011 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c"] Oct 30 14:45:57.448767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448751 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6f0597f7-23b7-4be4-af5e-4082e092826b" containerName="payload-processing" Oct 30 14:45:57.448820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448768 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0597f7-23b7-4be4-af5e-4082e092826b" containerName="payload-processing" Oct 30 14:45:57.448820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448786 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:57.448820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448791 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5076f0fd-cf19-46e1-8cb5-ab933b424acb" containerName="cleanup" Oct 30 14:45:57.448820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448800 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6c955a5d-f936-4233-a91b-7843ae0e5809" containerName="revoke-keys" Oct 30 14:45:57.448820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448804 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c955a5d-f936-4233-a91b-7843ae0e5809" containerName="revoke-keys" Oct 30 14:45:57.448820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448811 5824 cpu_manager.go:414] "containerMap: removing container" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:57.448820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448816 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448824 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448829 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448850 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448856 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab8d4c0-4ab9-45ed-bdc2-67d77c88393e" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448879 5824 cpu_manager.go:414] "containerMap: removing container" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448883 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448894 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448898 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0041a35-bbe1-40b1-ac40-f28760c6ec17" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448912 5824 cpu_manager.go:414] "containerMap: removing container" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.448916 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feee7683-a6bb-47fb-b085-6a510fc1b993" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.449005 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3661e6e3-d3ff-4e1d-a171-cd8ab00132be" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.449016 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cc8620-529c-47e4-ad22-0d4f8e998bc0" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.449024 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="110d3615-218b-49be-b6b6-99062913a1df" containerName="cleanup" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.449033 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0597f7-23b7-4be4-af5e-4082e092826b" containerName="payload-processing" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.449039 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c955a5d-f936-4233-a91b-7843ae0e5809" containerName="revoke-keys" Oct 30 14:45:57.449039 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.449045 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5c08e1-364b-456a-9888-a26fdd1055cb" containerName="cleanup" Oct 30 14:45:57.449551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.449050 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acc1072-69bf-4a93-8428-90ba63f5dea8" containerName="cleanup" Oct 30 14:45:57.454378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.454354 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.470135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.470111 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c"] Oct 30 14:45:57.479628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.479604 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:45:57.479798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.479643 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:45:57.546851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.546819 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.547050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.547021 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.547183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.547165 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7gv\" (UniqueName: \"kubernetes.io/projected/c40aec71-9f6a-4bf0-b208-293e8c143308-kube-api-access-vp7gv\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.648115 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.648087 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.648322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.648175 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7gv\" (UniqueName: \"kubernetes.io/projected/c40aec71-9f6a-4bf0-b208-293e8c143308-kube-api-access-vp7gv\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.648322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.648296 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.648917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.648895 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.650556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.650524 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.674359 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.674331 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7gv\" (UniqueName: \"kubernetes.io/projected/c40aec71-9f6a-4bf0-b208-293e8c143308-kube-api-access-vp7gv\") pod \"maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:57.769529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:57.769464 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:45:58.077765 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:45:58.077738 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40aec71_9f6a_4bf0_b208_293e8c143308.slice/crio-5e9fad7766b28c38acde170634e702ee7c09c2619139c527ddba965a06ca72ab WatchSource:0}: Error finding container 5e9fad7766b28c38acde170634e702ee7c09c2619139c527ddba965a06ca72ab: Status 404 returned error can't find the container with id 5e9fad7766b28c38acde170634e702ee7c09c2619139c527ddba965a06ca72ab Oct 30 14:45:58.079269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:58.079243 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c"] Oct 30 14:45:58.197473 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:58.197445 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" event={"ID":"c40aec71-9f6a-4bf0-b208-293e8c143308","Type":"ContainerStarted","Data":"5e9fad7766b28c38acde170634e702ee7c09c2619139c527ddba965a06ca72ab"} Oct 30 14:45:59.206884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:59.206856 5824 generic.go:356] "Generic (PLEG): container finished" podID="c40aec71-9f6a-4bf0-b208-293e8c143308" containerID="3bf596b4fd6bd1a0cd3b18b19a41578ea2cd40bed85897f68f2b1d772fb18b21" exitCode=0 Oct 30 14:45:59.207288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:45:59.206943 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" event={"ID":"c40aec71-9f6a-4bf0-b208-293e8c143308","Type":"ContainerDied","Data":"3bf596b4fd6bd1a0cd3b18b19a41578ea2cd40bed85897f68f2b1d772fb18b21"} Oct 30 14:46:00.371871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.371841 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:46:00.489857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.489784 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-openshift-ca\") pod \"c40aec71-9f6a-4bf0-b208-293e8c143308\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " Oct 30 14:46:00.490004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.489924 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-xks-ca\") pod \"c40aec71-9f6a-4bf0-b208-293e8c143308\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " Oct 30 14:46:00.490004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.489960 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40aec71-9f6a-4bf0-b208-293e8c143308-kube-api-access-vp7gv\" (UniqueName: \"kubernetes.io/projected/c40aec71-9f6a-4bf0-b208-293e8c143308-kube-api-access-vp7gv\") pod \"c40aec71-9f6a-4bf0-b208-293e8c143308\" (UID: \"c40aec71-9f6a-4bf0-b208-293e8c143308\") " Oct 30 14:46:00.490181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.490153 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-openshift-ca" pod "c40aec71-9f6a-4bf0-b208-293e8c143308" (UID: "c40aec71-9f6a-4bf0-b208-293e8c143308"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:00.490717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.490695 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:00.492031 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.492000 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40aec71-9f6a-4bf0-b208-293e8c143308-kube-api-access-vp7gv" pod "c40aec71-9f6a-4bf0-b208-293e8c143308" (UID: "c40aec71-9f6a-4bf0-b208-293e8c143308"). InnerVolumeSpecName "kube-api-access-vp7gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:00.492031 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.492006 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-xks-ca" pod "c40aec71-9f6a-4bf0-b208-293e8c143308" (UID: "c40aec71-9f6a-4bf0-b208-293e8c143308"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:00.591647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.591618 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c40aec71-9f6a-4bf0-b208-293e8c143308-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:00.591647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:00.591643 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7gv\" (UniqueName: \"kubernetes.io/projected/c40aec71-9f6a-4bf0-b208-293e8c143308-kube-api-access-vp7gv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:01.224393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:01.224362 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" Oct 30 14:46:01.224567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:01.224397 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c" event={"ID":"c40aec71-9f6a-4bf0-b208-293e8c143308","Type":"ContainerDied","Data":"5e9fad7766b28c38acde170634e702ee7c09c2619139c527ddba965a06ca72ab"} Oct 30 14:46:01.224567 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:01.224471 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e9fad7766b28c38acde170634e702ee7c09c2619139c527ddba965a06ca72ab" Oct 30 14:46:03.148035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.148000 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt"] Oct 30 14:46:03.148807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.148790 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c40aec71-9f6a-4bf0-b208-293e8c143308" containerName="revoke-keys" Oct 30 14:46:03.148868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.148808 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40aec71-9f6a-4bf0-b208-293e8c143308" containerName="revoke-keys" Oct 30 14:46:03.148992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.148979 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40aec71-9f6a-4bf0-b208-293e8c143308" containerName="revoke-keys" Oct 30 14:46:03.153054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.153036 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.171940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.171903 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt"] Oct 30 14:46:03.180179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.180152 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:46:03.180372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.180322 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:46:03.226485 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.226458 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgm97\" (UniqueName: \"kubernetes.io/projected/1ecb7561-c998-4ecf-8f24-d697654a3161-kube-api-access-kgm97\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.226637 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.226589 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.226686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.226667 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.328393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.328363 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.328542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.328410 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.328641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.328627 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgm97\" (UniqueName: \"kubernetes.io/projected/1ecb7561-c998-4ecf-8f24-d697654a3161-kube-api-access-kgm97\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.329117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.329094 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.330648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.330622 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.354787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.354768 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgm97\" (UniqueName: \"kubernetes.io/projected/1ecb7561-c998-4ecf-8f24-d697654a3161-kube-api-access-kgm97\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.472886 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.472860 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:03.641731 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:03.641678 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:46:03.778798 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:03.778760 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecb7561_c998_4ecf_8f24_d697654a3161.slice/crio-2f2c0de27178d8cdfbea950d86d7205db4868d41067377e1cf0814fde9aad89c WatchSource:0}: Error finding container 2f2c0de27178d8cdfbea950d86d7205db4868d41067377e1cf0814fde9aad89c: Status 404 returned error can't find the container with id 2f2c0de27178d8cdfbea950d86d7205db4868d41067377e1cf0814fde9aad89c Oct 30 14:46:03.781432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:03.781406 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt"] Oct 30 14:46:04.248635 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:04.248601 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" event={"ID":"1ecb7561-c998-4ecf-8f24-d697654a3161","Type":"ContainerStarted","Data":"35b8ad2dbbe45d29c835ef95051b460783d0a414279db0752f7a22fa3dea8f6c"} Oct 30 14:46:04.248987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:04.248649 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" event={"ID":"1ecb7561-c998-4ecf-8f24-d697654a3161","Type":"ContainerStarted","Data":"2f2c0de27178d8cdfbea950d86d7205db4868d41067377e1cf0814fde9aad89c"} Oct 30 14:46:04.293640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:04.293588 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" podStartSLOduration=1.293575546 podStartE2EDuration="1.293575546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:04.292496054 +0000 UTC m=+3394.093001496" watchObservedRunningTime="2026-10-30 14:46:04.293575546 +0000 UTC m=+3394.094080988" Oct 30 14:46:05.260335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:05.260300 5824 generic.go:356] "Generic (PLEG): container finished" podID="1ecb7561-c998-4ecf-8f24-d697654a3161" containerID="35b8ad2dbbe45d29c835ef95051b460783d0a414279db0752f7a22fa3dea8f6c" exitCode=0 Oct 30 14:46:05.262715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:05.260387 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" event={"ID":"1ecb7561-c998-4ecf-8f24-d697654a3161","Type":"ContainerDied","Data":"35b8ad2dbbe45d29c835ef95051b460783d0a414279db0752f7a22fa3dea8f6c"} Oct 30 14:46:05.505813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:05.504112 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm"] Oct 30 14:46:05.509739 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:05.509704 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a1854b-vqdsm"] Oct 30 14:46:06.216974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.216941 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj"] Oct 30 14:46:06.221438 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.221419 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.241757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.241732 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj"] Oct 30 14:46:06.298511 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.298475 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.298881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.298854 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.298929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.298897 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfhs\" (UniqueName: \"kubernetes.io/projected/267196fa-7db1-4e88-9e2a-92dc46b83cd4-kube-api-access-jwfhs\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.321477 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.321437 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c955a5d-f936-4233-a91b-7843ae0e5809" path="/var/lib/kubelet/pods/6c955a5d-f936-4233-a91b-7843ae0e5809/volumes" Oct 30 14:46:06.400702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.400670 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.400702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.400704 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfhs\" (UniqueName: \"kubernetes.io/projected/267196fa-7db1-4e88-9e2a-92dc46b83cd4-kube-api-access-jwfhs\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.401009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.400774 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.401572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.401469 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.403391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.403368 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.433048 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.433023 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfhs\" (UniqueName: \"kubernetes.io/projected/267196fa-7db1-4e88-9e2a-92dc46b83cd4-kube-api-access-jwfhs\") pod \"maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.436946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.436927 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:06.502044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.501966 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecb7561-c998-4ecf-8f24-d697654a3161-kube-api-access-kgm97\" (UniqueName: \"kubernetes.io/projected/1ecb7561-c998-4ecf-8f24-d697654a3161-kube-api-access-kgm97\") pod \"1ecb7561-c998-4ecf-8f24-d697654a3161\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " Oct 30 14:46:06.502212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.502118 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-openshift-ca\") pod \"1ecb7561-c998-4ecf-8f24-d697654a3161\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " Oct 30 14:46:06.502565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.502533 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-openshift-ca" pod "1ecb7561-c998-4ecf-8f24-d697654a3161" (UID: "1ecb7561-c998-4ecf-8f24-d697654a3161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:06.502694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.502669 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-xks-ca\") pod \"1ecb7561-c998-4ecf-8f24-d697654a3161\" (UID: \"1ecb7561-c998-4ecf-8f24-d697654a3161\") " Oct 30 14:46:06.503418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.503395 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:06.504583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.504558 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-xks-ca" pod "1ecb7561-c998-4ecf-8f24-d697654a3161" (UID: "1ecb7561-c998-4ecf-8f24-d697654a3161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:06.504916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.504887 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecb7561-c998-4ecf-8f24-d697654a3161-kube-api-access-kgm97" pod "1ecb7561-c998-4ecf-8f24-d697654a3161" (UID: "1ecb7561-c998-4ecf-8f24-d697654a3161"). InnerVolumeSpecName "kube-api-access-kgm97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:06.538527 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.538505 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:06.606376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.606343 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgm97\" (UniqueName: \"kubernetes.io/projected/1ecb7561-c998-4ecf-8f24-d697654a3161-kube-api-access-kgm97\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:06.606376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.606373 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecb7561-c998-4ecf-8f24-d697654a3161-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:06.847785 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:06.847754 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267196fa_7db1_4e88_9e2a_92dc46b83cd4.slice/crio-70bd520d9f15819418287ed2d433e028853b9452ed3278e619251b4bcfdbe6a6 WatchSource:0}: Error finding container 70bd520d9f15819418287ed2d433e028853b9452ed3278e619251b4bcfdbe6a6: Status 404 returned error can't find the container with id 70bd520d9f15819418287ed2d433e028853b9452ed3278e619251b4bcfdbe6a6 Oct 30 14:46:06.849708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:06.849662 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj"] Oct 30 14:46:07.254883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.254852 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md"] Oct 30 14:46:07.255061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.255031 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" podUID="02489a04-3f89-490e-a290-5608d5c61dd6" containerName="maas-api" containerID="cri-o://aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0" gracePeriod=30 Oct 30 14:46:07.286793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.286731 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" event={"ID":"267196fa-7db1-4e88-9e2a-92dc46b83cd4","Type":"ContainerStarted","Data":"10928c60e68a97155493e4579c5c58484064cc2f7795ec620ec9856199ef8ad6"} Oct 30 14:46:07.287082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.287041 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" event={"ID":"267196fa-7db1-4e88-9e2a-92dc46b83cd4","Type":"ContainerStarted","Data":"70bd520d9f15819418287ed2d433e028853b9452ed3278e619251b4bcfdbe6a6"} Oct 30 14:46:07.288559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.288537 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" event={"ID":"1ecb7561-c998-4ecf-8f24-d697654a3161","Type":"ContainerDied","Data":"2f2c0de27178d8cdfbea950d86d7205db4868d41067377e1cf0814fde9aad89c"} Oct 30 14:46:07.289156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.288575 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2c0de27178d8cdfbea950d86d7205db4868d41067377e1cf0814fde9aad89c" Oct 30 14:46:07.289156 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.288617 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt" Oct 30 14:46:07.373847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.373801 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" podStartSLOduration=1.3737859239999999 podStartE2EDuration="1.373785924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:07.370258053 +0000 UTC m=+3397.170763494" watchObservedRunningTime="2026-10-30 14:46:07.373785924 +0000 UTC m=+3397.174291367" Oct 30 14:46:07.619790 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.619669 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:46:07.722414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.721871 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02489a04-3f89-490e-a290-5608d5c61dd6-kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/02489a04-3f89-490e-a290-5608d5c61dd6-kube-api-access-nj5ps\") pod \"02489a04-3f89-490e-a290-5608d5c61dd6\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " Oct 30 14:46:07.722414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.721935 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02489a04-3f89-490e-a290-5608d5c61dd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02489a04-3f89-490e-a290-5608d5c61dd6-openshift-service-ca\") pod \"02489a04-3f89-490e-a290-5608d5c61dd6\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " Oct 30 14:46:07.722414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.722015 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls\") pod \"02489a04-3f89-490e-a290-5608d5c61dd6\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " Oct 30 14:46:07.722414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.722124 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-metrics-certs\") pod \"02489a04-3f89-490e-a290-5608d5c61dd6\" (UID: \"02489a04-3f89-490e-a290-5608d5c61dd6\") " Oct 30 14:46:07.728625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.728573 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02489a04-3f89-490e-a290-5608d5c61dd6-openshift-service-ca" pod "02489a04-3f89-490e-a290-5608d5c61dd6" (UID: "02489a04-3f89-490e-a290-5608d5c61dd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:07.741798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.741767 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02489a04-3f89-490e-a290-5608d5c61dd6-kube-api-access-nj5ps" pod "02489a04-3f89-490e-a290-5608d5c61dd6" (UID: "02489a04-3f89-490e-a290-5608d5c61dd6"). InnerVolumeSpecName "kube-api-access-nj5ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:07.743076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.743046 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-metrics-certs" pod "02489a04-3f89-490e-a290-5608d5c61dd6" (UID: "02489a04-3f89-490e-a290-5608d5c61dd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:07.743750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.743721 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls" pod "02489a04-3f89-490e-a290-5608d5c61dd6" (UID: "02489a04-3f89-490e-a290-5608d5c61dd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:07.837215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.836889 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5ps\" (UniqueName: \"kubernetes.io/projected/02489a04-3f89-490e-a290-5608d5c61dd6-kube-api-access-nj5ps\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:07.837215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.836929 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02489a04-3f89-490e-a290-5608d5c61dd6-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:07.837215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.836946 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:07.837215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:07.836961 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02489a04-3f89-490e-a290-5608d5c61dd6-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:08.134863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.134784 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst"] Oct 30 14:46:08.140401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.140379 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-ca229c-zcpst"] Oct 30 14:46:08.297520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.297476 5824 generic.go:356] "Generic (PLEG): container finished" podID="267196fa-7db1-4e88-9e2a-92dc46b83cd4" containerID="10928c60e68a97155493e4579c5c58484064cc2f7795ec620ec9856199ef8ad6" exitCode=0 Oct 30 14:46:08.297722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.297562 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" event={"ID":"267196fa-7db1-4e88-9e2a-92dc46b83cd4","Type":"ContainerDied","Data":"10928c60e68a97155493e4579c5c58484064cc2f7795ec620ec9856199ef8ad6"} Oct 30 14:46:08.298975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.298947 5824 generic.go:356] "Generic (PLEG): container finished" podID="02489a04-3f89-490e-a290-5608d5c61dd6" containerID="aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0" exitCode=0 Oct 30 14:46:08.299117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.299005 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" Oct 30 14:46:08.299117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.299030 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" event={"ID":"02489a04-3f89-490e-a290-5608d5c61dd6","Type":"ContainerDied","Data":"aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0"} Oct 30 14:46:08.299117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.299088 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md" event={"ID":"02489a04-3f89-490e-a290-5608d5c61dd6","Type":"ContainerDied","Data":"7b278435d81af5c21c12a820e8514a44ca10d1117cad3c859693dea768aa5822"} Oct 30 14:46:08.299117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.299115 5824 scope.go:122] "RemoveContainer" containerID="aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0" Oct 30 14:46:08.310721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.310705 5824 scope.go:122] "RemoveContainer" containerID="aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0" Oct 30 14:46:08.311013 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:08.310994 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0\": container with ID starting with aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0 not found: ID does not exist" containerID="aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0" Oct 30 14:46:08.311066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.311022 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0"} err="failed to get container status \"aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0\": rpc error: code = NotFound desc = could not find container \"aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0\": container with ID starting with aaa83d5ca62801059a8c59d5cfe122732d0f59452055818b3df3e97dc49b87b0 not found: ID does not exist" Oct 30 14:46:08.319250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.319177 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba98114-4665-46bf-a947-5a86f9897a3a" path="/var/lib/kubelet/pods/7ba98114-4665-46bf-a947-5a86f9897a3a/volumes" Oct 30 14:46:08.393076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.393000 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md"] Oct 30 14:46:08.400249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:08.400218 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca229c-85d6c4f945-t92md"] Oct 30 14:46:09.456456 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.456433 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:09.558541 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.558473 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267196fa-7db1-4e88-9e2a-92dc46b83cd4-kube-api-access-jwfhs\" (UniqueName: \"kubernetes.io/projected/267196fa-7db1-4e88-9e2a-92dc46b83cd4-kube-api-access-jwfhs\") pod \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " Oct 30 14:46:09.558692 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.558674 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-openshift-ca\") pod \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " Oct 30 14:46:09.558737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.558710 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-xks-ca\") pod \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\" (UID: \"267196fa-7db1-4e88-9e2a-92dc46b83cd4\") " Oct 30 14:46:09.559008 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.558978 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-openshift-ca" pod "267196fa-7db1-4e88-9e2a-92dc46b83cd4" (UID: "267196fa-7db1-4e88-9e2a-92dc46b83cd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:09.559422 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.559402 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:09.560587 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.560566 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267196fa-7db1-4e88-9e2a-92dc46b83cd4-kube-api-access-jwfhs" pod "267196fa-7db1-4e88-9e2a-92dc46b83cd4" (UID: "267196fa-7db1-4e88-9e2a-92dc46b83cd4"). InnerVolumeSpecName "kube-api-access-jwfhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:09.560672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.560618 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-xks-ca" pod "267196fa-7db1-4e88-9e2a-92dc46b83cd4" (UID: "267196fa-7db1-4e88-9e2a-92dc46b83cd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:09.660610 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.660584 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267196fa-7db1-4e88-9e2a-92dc46b83cd4-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:09.660610 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:09.660608 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfhs\" (UniqueName: \"kubernetes.io/projected/267196fa-7db1-4e88-9e2a-92dc46b83cd4-kube-api-access-jwfhs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:10.317954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:10.317926 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" Oct 30 14:46:10.318625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:10.318601 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02489a04-3f89-490e-a290-5608d5c61dd6" path="/var/lib/kubelet/pods/02489a04-3f89-490e-a290-5608d5c61dd6/volumes" Oct 30 14:46:10.319084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:10.319068 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj" event={"ID":"267196fa-7db1-4e88-9e2a-92dc46b83cd4","Type":"ContainerDied","Data":"70bd520d9f15819418287ed2d433e028853b9452ed3278e619251b4bcfdbe6a6"} Oct 30 14:46:10.319136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:10.319102 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70bd520d9f15819418287ed2d433e028853b9452ed3278e619251b4bcfdbe6a6" Oct 30 14:46:11.039174 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:11.039142 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz"] Oct 30 14:46:11.052043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:11.052010 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p"] Oct 30 14:46:11.065126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:11.065101 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl"] Oct 30 14:46:14.111361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.111327 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw"] Oct 30 14:46:14.112082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112065 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1ecb7561-c998-4ecf-8f24-d697654a3161" containerName="revoke-keys" Oct 30 14:46:14.112147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112083 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecb7561-c998-4ecf-8f24-d697654a3161" containerName="revoke-keys" Oct 30 14:46:14.112147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112096 5824 cpu_manager.go:414] "containerMap: removing container" podUID="267196fa-7db1-4e88-9e2a-92dc46b83cd4" containerName="revoke-keys" Oct 30 14:46:14.112147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112101 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267196fa-7db1-4e88-9e2a-92dc46b83cd4" containerName="revoke-keys" Oct 30 14:46:14.112147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112108 5824 cpu_manager.go:414] "containerMap: removing container" podUID="02489a04-3f89-490e-a290-5608d5c61dd6" containerName="maas-api" Oct 30 14:46:14.112147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112113 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02489a04-3f89-490e-a290-5608d5c61dd6" containerName="maas-api" Oct 30 14:46:14.112373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112228 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecb7561-c998-4ecf-8f24-d697654a3161" containerName="revoke-keys" Oct 30 14:46:14.112373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112240 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="267196fa-7db1-4e88-9e2a-92dc46b83cd4" containerName="revoke-keys" Oct 30 14:46:14.112373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.112246 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="02489a04-3f89-490e-a290-5608d5c61dd6" containerName="maas-api" Oct 30 14:46:14.117061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.117044 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.135587 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.135564 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:46:14.135948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.135922 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:46:14.137011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.136991 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw"] Oct 30 14:46:14.231436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.231407 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.231581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.231549 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.231634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.231619 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvjs\" (UniqueName: \"kubernetes.io/projected/b7123872-3925-491b-ad8c-075c3397f3af-kube-api-access-pfvjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.332342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.332313 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.332480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.332395 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.332480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.332425 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvjs\" (UniqueName: \"kubernetes.io/projected/b7123872-3925-491b-ad8c-075c3397f3af-kube-api-access-pfvjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.332994 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.332975 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.334618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.334601 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.358372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.358347 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvjs\" (UniqueName: \"kubernetes.io/projected/b7123872-3925-491b-ad8c-075c3397f3af-kube-api-access-pfvjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.431581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.431550 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:14.751266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:14.751233 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw"] Oct 30 14:46:14.752124 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:14.752092 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7123872_3925_491b_ad8c_075c3397f3af.slice/crio-c7e87c21ef89448ef03f66eb606c3d045fe8bc82a0d8e942904b507a67636bd0 WatchSource:0}: Error finding container c7e87c21ef89448ef03f66eb606c3d045fe8bc82a0d8e942904b507a67636bd0: Status 404 returned error can't find the container with id c7e87c21ef89448ef03f66eb606c3d045fe8bc82a0d8e942904b507a67636bd0 Oct 30 14:46:15.355884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:15.355794 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" event={"ID":"b7123872-3925-491b-ad8c-075c3397f3af","Type":"ContainerStarted","Data":"0aabf738ec426a46e9edd0af8bf8cb4864b28e042ea033139c0198051438d028"} Oct 30 14:46:15.355884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:15.355857 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" event={"ID":"b7123872-3925-491b-ad8c-075c3397f3af","Type":"ContainerStarted","Data":"c7e87c21ef89448ef03f66eb606c3d045fe8bc82a0d8e942904b507a67636bd0"} Oct 30 14:46:15.404011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:15.403955 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" podStartSLOduration=1.403940881 podStartE2EDuration="1.403940881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:15.401433425 +0000 UTC m=+3405.201938878" watchObservedRunningTime="2026-10-30 14:46:15.403940881 +0000 UTC m=+3405.204446322" Oct 30 14:46:16.377457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:16.377411 5824 generic.go:356] "Generic (PLEG): container finished" podID="b7123872-3925-491b-ad8c-075c3397f3af" containerID="0aabf738ec426a46e9edd0af8bf8cb4864b28e042ea033139c0198051438d028" exitCode=0 Oct 30 14:46:16.377880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:16.377486 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" event={"ID":"b7123872-3925-491b-ad8c-075c3397f3af","Type":"ContainerDied","Data":"0aabf738ec426a46e9edd0af8bf8cb4864b28e042ea033139c0198051438d028"} Oct 30 14:46:16.498437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:16.498406 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff"] Oct 30 14:46:16.498906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:16.498845 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" containerID="cri-o://a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269" gracePeriod=30 Oct 30 14:46:16.752426 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:16.752388 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9"] Oct 30 14:46:17.660581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.660208 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:17.814302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.814265 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-openshift-ca\") pod \"b7123872-3925-491b-ad8c-075c3397f3af\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " Oct 30 14:46:17.814478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.814347 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7123872-3925-491b-ad8c-075c3397f3af-kube-api-access-pfvjs\" (UniqueName: \"kubernetes.io/projected/b7123872-3925-491b-ad8c-075c3397f3af-kube-api-access-pfvjs\") pod \"b7123872-3925-491b-ad8c-075c3397f3af\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " Oct 30 14:46:17.814569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.814556 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-xks-ca\") pod \"b7123872-3925-491b-ad8c-075c3397f3af\" (UID: \"b7123872-3925-491b-ad8c-075c3397f3af\") " Oct 30 14:46:17.814643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.814616 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-openshift-ca" pod "b7123872-3925-491b-ad8c-075c3397f3af" (UID: "b7123872-3925-491b-ad8c-075c3397f3af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:17.815096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.815073 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:17.816560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.816532 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-xks-ca" pod "b7123872-3925-491b-ad8c-075c3397f3af" (UID: "b7123872-3925-491b-ad8c-075c3397f3af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:17.816676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.816650 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7123872-3925-491b-ad8c-075c3397f3af-kube-api-access-pfvjs" pod "b7123872-3925-491b-ad8c-075c3397f3af" (UID: "b7123872-3925-491b-ad8c-075c3397f3af"). InnerVolumeSpecName "kube-api-access-pfvjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:17.916158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.916095 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfvjs\" (UniqueName: \"kubernetes.io/projected/b7123872-3925-491b-ad8c-075c3397f3af-kube-api-access-pfvjs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:17.916158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:17.916126 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7123872-3925-491b-ad8c-075c3397f3af-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:18.394273 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:18.394243 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" Oct 30 14:46:18.394403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:18.394249 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw" event={"ID":"b7123872-3925-491b-ad8c-075c3397f3af","Type":"ContainerDied","Data":"c7e87c21ef89448ef03f66eb606c3d045fe8bc82a0d8e942904b507a67636bd0"} Oct 30 14:46:18.394403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:18.394364 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e87c21ef89448ef03f66eb606c3d045fe8bc82a0d8e942904b507a67636bd0" Oct 30 14:46:18.601276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:18.601202 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc"] Oct 30 14:46:18.601422 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:18.601395 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" containerID="cri-o://73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd" gracePeriod=30 Oct 30 14:46:21.753389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.753335 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" podUID="137c45cd-711f-46e6-a808-7e396102fb7f" containerName="payload-pre-processing" containerID="cri-o://ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f" gracePeriod=25 Oct 30 14:46:21.789558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.789507 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:46:21.887848 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.887816 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-token\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-token\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.887898 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-socket\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.887962 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-data\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.888078 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-envoy\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.888119 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-podinfo\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.888147 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-kube-api-access-5pvlb\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-kube-api-access-5pvlb\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.888202 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-certs\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.888239 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-socket" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:21.888397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.888285 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3798ac-47a0-4fae-9b07-483c1ead2451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3798ac-47a0-4fae-9b07-483c1ead2451-istiod-ca-cert\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.888350 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-credential-socket\") pod \"ef3798ac-47a0-4fae-9b07-483c1ead2451\" (UID: \"ef3798ac-47a0-4fae-9b07-483c1ead2451\") " Oct 30 14:46:21.888954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.888921 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:21.889146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.889117 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-credential-socket" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:21.889644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.889615 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-certs" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:21.889810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.889787 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3798ac-47a0-4fae-9b07-483c1ead2451-istiod-ca-cert" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:21.889880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.889863 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-data" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:21.890819 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.890789 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-podinfo" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:46:21.891432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.891398 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-envoy" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:21.891629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.891607 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-token" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:21.893083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.893056 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-kube-api-access-5pvlb" pod "ef3798ac-47a0-4fae-9b07-483c1ead2451" (UID: "ef3798ac-47a0-4fae-9b07-483c1ead2451"). InnerVolumeSpecName "kube-api-access-5pvlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:21.990800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.990769 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:21.990800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.990796 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:21.990800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.990805 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:21.991025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.990814 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pvlb\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-kube-api-access-5pvlb\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:21.991025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.990824 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:21.991025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.990833 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3798ac-47a0-4fae-9b07-483c1ead2451-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:21.991025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.990842 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3798ac-47a0-4fae-9b07-483c1ead2451-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:21.991025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:21.990849 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3798ac-47a0-4fae-9b07-483c1ead2451-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:22.424232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.424174 5824 generic.go:356] "Generic (PLEG): container finished" podID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerID="a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269" exitCode=0 Oct 30 14:46:22.424357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.424248 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" Oct 30 14:46:22.424357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.424265 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" event={"ID":"ef3798ac-47a0-4fae-9b07-483c1ead2451","Type":"ContainerDied","Data":"a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269"} Oct 30 14:46:22.424357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.424314 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff" event={"ID":"ef3798ac-47a0-4fae-9b07-483c1ead2451","Type":"ContainerDied","Data":"36de1f4ad8d862d4c67575cbbacd6c24b410ad92ee75ce197e5e316ce74d925e"} Oct 30 14:46:22.424357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.424335 5824 scope.go:122] "RemoveContainer" containerID="a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269" Oct 30 14:46:22.435120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.435104 5824 scope.go:122] "RemoveContainer" containerID="a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269" Oct 30 14:46:22.435427 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:22.435403 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269\": container with ID starting with a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269 not found: ID does not exist" containerID="a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269" Oct 30 14:46:22.435526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.435433 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269"} err="failed to get container status \"a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269\": rpc error: code = NotFound desc = could not find container \"a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269\": container with ID starting with a9d9666164f0b5ae601d341b9792d50b1c4d68f1c77341b4fe9a6f43b8189269 not found: ID does not exist" Oct 30 14:46:22.498376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.498357 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff"] Oct 30 14:46:22.512201 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:22.512171 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca229c-openshift-default-549cb656c9-fbmff"] Oct 30 14:46:23.891637 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.891610 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:46:23.918507 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918475 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-data\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.918626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918517 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-certs\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.918626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918582 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-credential-socket\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.918722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918654 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-envoy\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.918722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918707 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-kube-api-access-sw6zb\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-kube-api-access-sw6zb\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.918808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918756 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-data" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:23.918808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918763 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-podinfo\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.918808 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918796 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-certs" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:23.918921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918842 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-token\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-token\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.918969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918938 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-credential-socket" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:23.919013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.918968 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istiod-ca-cert\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.919067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.919017 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-socket\") pod \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\" (UID: \"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0\") " Oct 30 14:46:23.921074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.919642 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istiod-ca-cert" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:23.921074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.919734 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-socket" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:23.921074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.919780 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:23.921074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.919807 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:23.921074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.919822 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:23.921074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.919861 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:23.921427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.921265 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-envoy" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:23.922344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.922316 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-podinfo" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:46:23.922569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.922542 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-kube-api-access-sw6zb" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "kube-api-access-sw6zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:23.922804 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:23.922783 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-token" pod "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" (UID: "1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:24.021099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.021064 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:24.021099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.021095 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:24.021346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.021106 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw6zb\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-kube-api-access-sw6zb\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:24.021346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.021118 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:24.021346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.021128 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:24.222208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.222167 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q"] Oct 30 14:46:24.222393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.222374 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" containerID="cri-o://f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4" gracePeriod=30 Oct 30 14:46:24.317951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.317918 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" path="/var/lib/kubelet/pods/ef3798ac-47a0-4fae-9b07-483c1ead2451/volumes" Oct 30 14:46:24.457865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.457829 5824 generic.go:356] "Generic (PLEG): container finished" podID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerID="73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd" exitCode=0 Oct 30 14:46:24.458061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.457892 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" Oct 30 14:46:24.458061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.457905 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" event={"ID":"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0","Type":"ContainerDied","Data":"73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd"} Oct 30 14:46:24.458061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.457992 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc" event={"ID":"1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0","Type":"ContainerDied","Data":"e91341e55f3b4874d9b0b874b2beb878e6c7fd968f41d39349eeebec29c1bf48"} Oct 30 14:46:24.458061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.458020 5824 scope.go:122] "RemoveContainer" containerID="73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd" Oct 30 14:46:24.468696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.468674 5824 scope.go:122] "RemoveContainer" containerID="73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd" Oct 30 14:46:24.468987 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:24.468965 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd\": container with ID starting with 73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd not found: ID does not exist" containerID="73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd" Oct 30 14:46:24.469030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.468994 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd"} err="failed to get container status \"73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd\": rpc error: code = NotFound desc = could not find container \"73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd\": container with ID starting with 73c47f2436b643bbd42c8b78562f5acbbade94a44e551967a7cb792821be2edd not found: ID does not exist" Oct 30 14:46:24.518881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.518813 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc"] Oct 30 14:46:24.524664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:24.524641 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-396d70-openshift-default-59bc6b4c65-fwmdc"] Oct 30 14:46:26.060398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.060360 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" containerID="cri-o://af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775" gracePeriod=45 Oct 30 14:46:26.073728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.073695 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" containerID="cri-o://c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c" gracePeriod=45 Oct 30 14:46:26.090000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.089962 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" containerID="cri-o://d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006" gracePeriod=45 Oct 30 14:46:26.330487 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.330454 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" path="/var/lib/kubelet/pods/1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0/volumes" Oct 30 14:46:26.427337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.427261 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:46:26.430577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.430509 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:46:26.434802 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.434778 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:46:26.478314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.478053 5824 generic.go:356] "Generic (PLEG): container finished" podID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerID="af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775" exitCode=0 Oct 30 14:46:26.478314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.478245 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" event={"ID":"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff","Type":"ContainerDied","Data":"af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775"} Oct 30 14:46:26.478314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.478281 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" event={"ID":"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff","Type":"ContainerDied","Data":"a5bb95494337078fe4866c0e3920c2dc74ecb599443c6478275b9d37a8aa8c24"} Oct 30 14:46:26.478314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.478303 5824 scope.go:122] "RemoveContainer" containerID="af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775" Oct 30 14:46:26.478651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.478411 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz" Oct 30 14:46:26.481460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.481432 5824 generic.go:356] "Generic (PLEG): container finished" podID="2230a84a-9c4e-413d-8612-87844196be0e" containerID="d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006" exitCode=0 Oct 30 14:46:26.481622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.481473 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" event={"ID":"2230a84a-9c4e-413d-8612-87844196be0e","Type":"ContainerDied","Data":"d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006"} Oct 30 14:46:26.481622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.481514 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" event={"ID":"2230a84a-9c4e-413d-8612-87844196be0e","Type":"ContainerDied","Data":"94cd90bd8e9f56d256fcb550a27784b7352ec84de25ef0c7c4739d1245876857"} Oct 30 14:46:26.481794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.481771 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl" Oct 30 14:46:26.490692 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.490664 5824 generic.go:356] "Generic (PLEG): container finished" podID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerID="c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c" exitCode=0 Oct 30 14:46:26.490871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.490706 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" Oct 30 14:46:26.490871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.490796 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" event={"ID":"c45f55e6-f5fe-4103-88a5-3684bfd25ccf","Type":"ContainerDied","Data":"c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c"} Oct 30 14:46:26.490871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.490822 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p" event={"ID":"c45f55e6-f5fe-4103-88a5-3684bfd25ccf","Type":"ContainerDied","Data":"901f31ca7b0389834b3b9f3e6b666230bda5b54c8cc1618afe72c68b52742a43"} Oct 30 14:46:26.500261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.500126 5824 scope.go:122] "RemoveContainer" containerID="af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775" Oct 30 14:46:26.500675 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:26.500648 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775\": container with ID starting with af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775 not found: ID does not exist" containerID="af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775" Oct 30 14:46:26.500778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.500690 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775"} err="failed to get container status \"af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775\": rpc error: code = NotFound desc = could not find container \"af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775\": container with ID starting with af047cd966f77bbc986c48cca0626328a8716c0aa384b59f89fd39d71db89775 not found: ID does not exist" Oct 30 14:46:26.500778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.500715 5824 scope.go:122] "RemoveContainer" containerID="d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006" Oct 30 14:46:26.514062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.514041 5824 scope.go:122] "RemoveContainer" containerID="d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006" Oct 30 14:46:26.514476 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:26.514453 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006\": container with ID starting with d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006 not found: ID does not exist" containerID="d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006" Oct 30 14:46:26.514588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.514483 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006"} err="failed to get container status \"d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006\": rpc error: code = NotFound desc = could not find container \"d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006\": container with ID starting with d35e154e04dbae500aec19e00d9a3932b2e5debbcaee031b11bd937ccf754006 not found: ID does not exist" Oct 30 14:46:26.514588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.514501 5824 scope.go:122] "RemoveContainer" containerID="c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c" Oct 30 14:46:26.526319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.526290 5824 scope.go:122] "RemoveContainer" containerID="c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c" Oct 30 14:46:26.526709 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:26.526680 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c\": container with ID starting with c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c not found: ID does not exist" containerID="c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c" Oct 30 14:46:26.526826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.526715 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c"} err="failed to get container status \"c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c\": rpc error: code = NotFound desc = could not find container \"c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c\": container with ID starting with c6a7c07a985cc6af22af563000d6fb1a791cf2b164dd717dc66355629230a17c not found: ID does not exist" Oct 30 14:46:26.557434 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557398 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tls-certs\" (UniqueName: \"kubernetes.io/secret/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tls-certs\") pod \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " Oct 30 14:46:26.557611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557442 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tmp-dir\") pod \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " Oct 30 14:46:26.557611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557472 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-model-cache\") pod \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " Oct 30 14:46:26.557611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557507 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-model-cache\") pod \"2230a84a-9c4e-413d-8612-87844196be0e\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " Oct 30 14:46:26.557611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557536 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-dshm\") pod \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " Oct 30 14:46:26.557611 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557606 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2230a84a-9c4e-413d-8612-87844196be0e-kube-api-access-sm7fh\" (UniqueName: \"kubernetes.io/projected/2230a84a-9c4e-413d-8612-87844196be0e-kube-api-access-sm7fh\") pod \"2230a84a-9c4e-413d-8612-87844196be0e\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " Oct 30 14:46:26.557912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557683 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tls-certs\") pod \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " Oct 30 14:46:26.557912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557719 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-dshm\") pod \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " Oct 30 14:46:26.557912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557707 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tmp-dir" pod "c45f55e6-f5fe-4103-88a5-3684bfd25ccf" (UID: "c45f55e6-f5fe-4103-88a5-3684bfd25ccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.557912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557750 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-home\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-home\") pod \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " Oct 30 14:46:26.557912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557784 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2230a84a-9c4e-413d-8612-87844196be0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2230a84a-9c4e-413d-8612-87844196be0e-tls-certs\") pod \"2230a84a-9c4e-413d-8612-87844196be0e\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " Oct 30 14:46:26.557912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557840 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-dshm\") pod \"2230a84a-9c4e-413d-8612-87844196be0e\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " Oct 30 14:46:26.557912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557889 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-home\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-home\") pod \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " Oct 30 14:46:26.558263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.557932 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-model-cache\") pod \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " Oct 30 14:46:26.558263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558019 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-kube-api-access-mrrgf\" (UniqueName: \"kubernetes.io/projected/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-kube-api-access-mrrgf\") pod \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\" (UID: \"c45f55e6-f5fe-4103-88a5-3684bfd25ccf\") " Oct 30 14:46:26.558263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558031 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-model-cache" pod "c45f55e6-f5fe-4103-88a5-3684bfd25ccf" (UID: "c45f55e6-f5fe-4103-88a5-3684bfd25ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.558263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558092 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tmp-dir\") pod \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " Oct 30 14:46:26.558263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558138 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-tmp-dir\") pod \"2230a84a-9c4e-413d-8612-87844196be0e\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " Oct 30 14:46:26.558896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558290 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-home\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-home\") pod \"2230a84a-9c4e-413d-8612-87844196be0e\" (UID: \"2230a84a-9c4e-413d-8612-87844196be0e\") " Oct 30 14:46:26.558896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558346 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-kube-api-access-s5cpc\" (UniqueName: \"kubernetes.io/projected/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-kube-api-access-s5cpc\") pod \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\" (UID: \"20a4a5cf-d7f8-4243-a491-71c65e5ed8ff\") " Oct 30 14:46:26.558896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558411 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-home" pod "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" (UID: "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.558896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558541 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-model-cache" pod "2230a84a-9c4e-413d-8612-87844196be0e" (UID: "2230a84a-9c4e-413d-8612-87844196be0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.558896 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558778 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-home" pod "c45f55e6-f5fe-4103-88a5-3684bfd25ccf" (UID: "c45f55e6-f5fe-4103-88a5-3684bfd25ccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.559242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.558975 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tmp-dir" pod "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" (UID: "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.559242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559214 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-model-cache" pod "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" (UID: "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.559495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559468 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.559884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559498 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.559884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559547 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.559884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559565 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.559884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559578 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.559884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559590 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.559884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559603 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.560281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.559902 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-tmp-dir" pod "2230a84a-9c4e-413d-8612-87844196be0e" (UID: "2230a84a-9c4e-413d-8612-87844196be0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.560281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.560116 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-home" pod "2230a84a-9c4e-413d-8612-87844196be0e" (UID: "2230a84a-9c4e-413d-8612-87844196be0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.561442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.561404 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-kube-api-access-s5cpc" pod "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" (UID: "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff"). InnerVolumeSpecName "kube-api-access-s5cpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:26.562626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.562163 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2230a84a-9c4e-413d-8612-87844196be0e-kube-api-access-sm7fh" pod "2230a84a-9c4e-413d-8612-87844196be0e" (UID: "2230a84a-9c4e-413d-8612-87844196be0e"). InnerVolumeSpecName "kube-api-access-sm7fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:26.563479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.562890 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tls-certs" pod "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" (UID: "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:26.563686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.563660 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-dshm" pod "2230a84a-9c4e-413d-8612-87844196be0e" (UID: "2230a84a-9c4e-413d-8612-87844196be0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.563686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.563660 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2230a84a-9c4e-413d-8612-87844196be0e-tls-certs" pod "2230a84a-9c4e-413d-8612-87844196be0e" (UID: "2230a84a-9c4e-413d-8612-87844196be0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:26.564010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.563977 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tls-certs" pod "c45f55e6-f5fe-4103-88a5-3684bfd25ccf" (UID: "c45f55e6-f5fe-4103-88a5-3684bfd25ccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:26.564277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.564245 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-kube-api-access-mrrgf" pod "c45f55e6-f5fe-4103-88a5-3684bfd25ccf" (UID: "c45f55e6-f5fe-4103-88a5-3684bfd25ccf"). InnerVolumeSpecName "kube-api-access-mrrgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:26.564382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.564280 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-dshm" pod "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" (UID: "20a4a5cf-d7f8-4243-a491-71c65e5ed8ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.564382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.564302 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-dshm" pod "c45f55e6-f5fe-4103-88a5-3684bfd25ccf" (UID: "c45f55e6-f5fe-4103-88a5-3684bfd25ccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:26.660966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660889 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrgf\" (UniqueName: \"kubernetes.io/projected/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-kube-api-access-mrrgf\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.660966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660920 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.660966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660929 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.660966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660938 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5cpc\" (UniqueName: \"kubernetes.io/projected/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-kube-api-access-s5cpc\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.660966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660950 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.660966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660962 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c45f55e6-f5fe-4103-88a5-3684bfd25ccf-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.660966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660971 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7fh\" (UniqueName: \"kubernetes.io/projected/2230a84a-9c4e-413d-8612-87844196be0e-kube-api-access-sm7fh\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.661326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660980 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.661326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660990 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.661326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.660998 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2230a84a-9c4e-413d-8612-87844196be0e-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.661326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.661005 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2230a84a-9c4e-413d-8612-87844196be0e-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:26.832373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.832345 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz"] Oct 30 14:46:26.837379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.837345 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-2-w5-a7bb0c-kserve-7b57748d74-ghdzz"] Oct 30 14:46:26.875633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.875606 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl"] Oct 30 14:46:26.883552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.883530 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/premium-w5-a7bb0c-kserve-889887c7d-6s7tl"] Oct 30 14:46:26.922737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.922667 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p"] Oct 30 14:46:26.928386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:26.928358 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a7bb0c/distinct-w5-a7bb0c-kserve-5648d69f49-p279p"] Oct 30 14:46:27.232675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:27.232639 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" podUID="137c45cd-711f-46e6-a808-7e396102fb7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:46:28.318564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:28.318533 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" path="/var/lib/kubelet/pods/20a4a5cf-d7f8-4243-a491-71c65e5ed8ff/volumes" Oct 30 14:46:28.318975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:28.318960 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2230a84a-9c4e-413d-8612-87844196be0e" path="/var/lib/kubelet/pods/2230a84a-9c4e-413d-8612-87844196be0e/volumes" Oct 30 14:46:28.319539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:28.319518 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" path="/var/lib/kubelet/pods/c45f55e6-f5fe-4103-88a5-3684bfd25ccf/volumes" Oct 30 14:46:28.373651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:28.372448 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw"] Oct 30 14:46:28.386014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:28.385982 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4be240-8s4dw"] Oct 30 14:46:29.507027 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.506964 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:46:29.516364 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.516335 5824 generic.go:356] "Generic (PLEG): container finished" podID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerID="f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4" exitCode=0 Oct 30 14:46:29.516510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.516380 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" Oct 30 14:46:29.516510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.516401 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" event={"ID":"dd201164-e0c1-4544-9892-4353f4bfadcb","Type":"ContainerDied","Data":"f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4"} Oct 30 14:46:29.516510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.516464 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q" event={"ID":"dd201164-e0c1-4544-9892-4353f4bfadcb","Type":"ContainerDied","Data":"ead3c298b065f30f4f0e28f63fba6fa9470994d92ac93b3ce85a456b6eab5f3c"} Oct 30 14:46:29.516510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.516491 5824 scope.go:122] "RemoveContainer" containerID="f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4" Oct 30 14:46:29.526825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.526806 5824 scope.go:122] "RemoveContainer" containerID="f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4" Oct 30 14:46:29.527116 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:29.527094 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4\": container with ID starting with f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4 not found: ID does not exist" containerID="f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4" Oct 30 14:46:29.527323 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.527121 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4"} err="failed to get container status \"f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4\": rpc error: code = NotFound desc = could not find container \"f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4\": container with ID starting with f0b780115d50c838ad4da6d01d45d7efda28751e8aa656fd300daddb6f5260e4 not found: ID does not exist" Oct 30 14:46:29.610792 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.610757 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-kube-api-access-k7sx5\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-kube-api-access-k7sx5\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.610989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.610847 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-envoy\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.610989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.610899 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-data\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.610989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.610940 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-certs\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.610989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.610969 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-credential-socket\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.611243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611061 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-socket\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.611243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611094 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-podinfo\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.611366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611254 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd201164-e0c1-4544-9892-4353f4bfadcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd201164-e0c1-4544-9892-4353f4bfadcb-istiod-ca-cert\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.611366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611280 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-data" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:29.611366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611273 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-certs" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:29.611366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611325 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-token\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-token\") pod \"dd201164-e0c1-4544-9892-4353f4bfadcb\" (UID: \"dd201164-e0c1-4544-9892-4353f4bfadcb\") " Oct 30 14:46:29.611584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611416 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-credential-socket" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:29.611584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611456 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-socket" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:29.611667 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.611638 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd201164-e0c1-4544-9892-4353f4bfadcb-istiod-ca-cert" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:29.612059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.612031 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.612059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.612061 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.612297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.612074 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.612297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.612089 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.612297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.612102 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd201164-e0c1-4544-9892-4353f4bfadcb-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.613117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.613090 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-envoy" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:46:29.613344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.613315 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-podinfo" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:46:29.613478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.613455 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-token" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:29.613564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.613547 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-kube-api-access-k7sx5" pod "dd201164-e0c1-4544-9892-4353f4bfadcb" (UID: "dd201164-e0c1-4544-9892-4353f4bfadcb"). InnerVolumeSpecName "kube-api-access-k7sx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:29.713751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.713712 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.713751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.713746 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.713964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.713758 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.713964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.713773 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7sx5\" (UniqueName: \"kubernetes.io/projected/dd201164-e0c1-4544-9892-4353f4bfadcb-kube-api-access-k7sx5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:29.869814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.869774 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q"] Oct 30 14:46:29.883010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:29.882983 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0efe43-openshift-default-6b8ffb8d99-j5h4q"] Oct 30 14:46:30.318495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:30.318463 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7123872-3925-491b-ad8c-075c3397f3af" path="/var/lib/kubelet/pods/b7123872-3925-491b-ad8c-075c3397f3af/volumes" Oct 30 14:46:30.318930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:30.318915 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" path="/var/lib/kubelet/pods/dd201164-e0c1-4544-9892-4353f4bfadcb/volumes" Oct 30 14:46:32.340227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:32.340175 5824 scope.go:122] "RemoveContainer" containerID="a8f71f18891f49cc5ff91964ddc02b68ab3eb7db55e6fbe7ef8d4208011897be" Oct 30 14:46:32.353044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:32.353019 5824 scope.go:122] "RemoveContainer" containerID="83b94125a4f886b261ebe4c02fb0816e2c76031265c5edda0e2a8f25d5a3f9e4" Oct 30 14:46:33.640928 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:33.640889 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:46:36.631609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.631576 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr"] Oct 30 14:46:36.632249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632235 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b7123872-3925-491b-ad8c-075c3397f3af" containerName="revoke-keys" Oct 30 14:46:36.632249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632250 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7123872-3925-491b-ad8c-075c3397f3af" containerName="revoke-keys" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632261 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632268 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632290 5824 cpu_manager.go:414] "containerMap: removing container" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632295 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632307 5824 cpu_manager.go:414] "containerMap: removing container" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632312 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632319 5824 cpu_manager.go:414] "containerMap: removing container" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632323 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632334 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" Oct 30 14:46:36.632336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632338 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632348 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632353 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632462 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45f55e6-f5fe-4103-88a5-3684bfd25ccf" containerName="main" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632474 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7123872-3925-491b-ad8c-075c3397f3af" containerName="revoke-keys" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632482 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7cbdf7-c946-4b38-8716-4f52cc8bc9b0" containerName="istio-proxy" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632491 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="2230a84a-9c4e-413d-8612-87844196be0e" containerName="main" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632497 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd201164-e0c1-4544-9892-4353f4bfadcb" containerName="istio-proxy" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632502 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a4a5cf-d7f8-4243-a491-71c65e5ed8ff" containerName="main" Oct 30 14:46:36.632640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.632507 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3798ac-47a0-4fae-9b07-483c1ead2451" containerName="istio-proxy" Oct 30 14:46:36.635917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.635900 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.656593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.656556 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr"] Oct 30 14:46:36.737240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.737209 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:46:36.737412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.737292 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:46:36.813687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.813651 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5jml\" (UniqueName: \"kubernetes.io/projected/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-kube-api-access-c5jml\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.813849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.813757 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.813849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.813826 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.914880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.914812 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5jml\" (UniqueName: \"kubernetes.io/projected/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-kube-api-access-c5jml\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.914880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.914865 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.915107 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.914903 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.915579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.915556 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.917302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.917277 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.945973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.945947 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5jml\" (UniqueName: \"kubernetes.io/projected/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-kube-api-access-c5jml\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:36.949626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:36.949609 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:37.234122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:37.233018 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" podUID="137c45cd-711f-46e6-a808-7e396102fb7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:46:37.260265 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:37.260234 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cef0a4c_3529_4476_8ddf_f4e60e3f017e.slice/crio-0ece1e7f6caab77b78d7f5f97488d762cf5d1cb5393528b900eadf564fa3e818 WatchSource:0}: Error finding container 0ece1e7f6caab77b78d7f5f97488d762cf5d1cb5393528b900eadf564fa3e818: Status 404 returned error can't find the container with id 0ece1e7f6caab77b78d7f5f97488d762cf5d1cb5393528b900eadf564fa3e818 Oct 30 14:46:37.260739 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:37.260695 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr"] Oct 30 14:46:37.588740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:37.588711 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" event={"ID":"5cef0a4c-3529-4476-8ddf-f4e60e3f017e","Type":"ContainerStarted","Data":"ead6d75c066ccc671b43ac586586153db0295d4defdf05e090bc7cb95154d0f0"} Oct 30 14:46:37.588870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:37.588764 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" event={"ID":"5cef0a4c-3529-4476-8ddf-f4e60e3f017e","Type":"ContainerStarted","Data":"0ece1e7f6caab77b78d7f5f97488d762cf5d1cb5393528b900eadf564fa3e818"} Oct 30 14:46:37.642033 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:37.641982 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" podStartSLOduration=1.641968468 podStartE2EDuration="1.641968468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:37.64041353 +0000 UTC m=+3427.440918974" watchObservedRunningTime="2026-10-30 14:46:37.641968468 +0000 UTC m=+3427.442473953" Oct 30 14:46:38.597617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:38.597583 5824 generic.go:356] "Generic (PLEG): container finished" podID="5cef0a4c-3529-4476-8ddf-f4e60e3f017e" containerID="ead6d75c066ccc671b43ac586586153db0295d4defdf05e090bc7cb95154d0f0" exitCode=0 Oct 30 14:46:38.597768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:38.597663 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" event={"ID":"5cef0a4c-3529-4476-8ddf-f4e60e3f017e","Type":"ContainerDied","Data":"ead6d75c066ccc671b43ac586586153db0295d4defdf05e090bc7cb95154d0f0"} Oct 30 14:46:39.751554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.751524 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:39.851351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.849552 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-xks-ca\") pod \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " Oct 30 14:46:39.851351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.849767 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-openshift-ca\") pod \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " Oct 30 14:46:39.851351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.849805 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-kube-api-access-c5jml\" (UniqueName: \"kubernetes.io/projected/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-kube-api-access-c5jml\") pod \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\" (UID: \"5cef0a4c-3529-4476-8ddf-f4e60e3f017e\") " Oct 30 14:46:39.851351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.850276 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-openshift-ca" pod "5cef0a4c-3529-4476-8ddf-f4e60e3f017e" (UID: "5cef0a4c-3529-4476-8ddf-f4e60e3f017e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:39.851800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.851749 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:39.852933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.852889 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-kube-api-access-c5jml" pod "5cef0a4c-3529-4476-8ddf-f4e60e3f017e" (UID: "5cef0a4c-3529-4476-8ddf-f4e60e3f017e"). InnerVolumeSpecName "kube-api-access-c5jml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:39.853109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.853071 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-xks-ca" pod "5cef0a4c-3529-4476-8ddf-f4e60e3f017e" (UID: "5cef0a4c-3529-4476-8ddf-f4e60e3f017e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:39.953255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.953223 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5jml\" (UniqueName: \"kubernetes.io/projected/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-kube-api-access-c5jml\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:39.953255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:39.953248 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cef0a4c-3529-4476-8ddf-f4e60e3f017e-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:40.612132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:40.612067 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" Oct 30 14:46:40.612132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:40.612105 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr" event={"ID":"5cef0a4c-3529-4476-8ddf-f4e60e3f017e","Type":"ContainerDied","Data":"0ece1e7f6caab77b78d7f5f97488d762cf5d1cb5393528b900eadf564fa3e818"} Oct 30 14:46:40.612338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:40.612150 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ece1e7f6caab77b78d7f5f97488d762cf5d1cb5393528b900eadf564fa3e818" Oct 30 14:46:42.028101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.028037 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:46:42.076294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.076265 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137c45cd-711f-46e6-a808-7e396102fb7f-kube-api-access-t2mhb\" (UniqueName: \"kubernetes.io/projected/137c45cd-711f-46e6-a808-7e396102fb7f-kube-api-access-t2mhb\") pod \"137c45cd-711f-46e6-a808-7e396102fb7f\" (UID: \"137c45cd-711f-46e6-a808-7e396102fb7f\") " Oct 30 14:46:42.078336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.078309 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137c45cd-711f-46e6-a808-7e396102fb7f-kube-api-access-t2mhb" pod "137c45cd-711f-46e6-a808-7e396102fb7f" (UID: "137c45cd-711f-46e6-a808-7e396102fb7f"). InnerVolumeSpecName "kube-api-access-t2mhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:42.177475 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.177454 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137c45cd-711f-46e6-a808-7e396102fb7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c45cd-711f-46e6-a808-7e396102fb7f-plugins-config-volume\") pod \"137c45cd-711f-46e6-a808-7e396102fb7f\" (UID: \"137c45cd-711f-46e6-a808-7e396102fb7f\") " Oct 30 14:46:42.177768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.177745 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137c45cd-711f-46e6-a808-7e396102fb7f-plugins-config-volume" pod "137c45cd-711f-46e6-a808-7e396102fb7f" (UID: "137c45cd-711f-46e6-a808-7e396102fb7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:42.177975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.177962 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2mhb\" (UniqueName: \"kubernetes.io/projected/137c45cd-711f-46e6-a808-7e396102fb7f-kube-api-access-t2mhb\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:42.178020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.177977 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c45cd-711f-46e6-a808-7e396102fb7f-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:42.626362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.626286 5824 generic.go:356] "Generic (PLEG): container finished" podID="137c45cd-711f-46e6-a808-7e396102fb7f" containerID="ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f" exitCode=0 Oct 30 14:46:42.626362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.626349 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" Oct 30 14:46:42.626595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.626369 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" event={"ID":"137c45cd-711f-46e6-a808-7e396102fb7f","Type":"ContainerDied","Data":"ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f"} Oct 30 14:46:42.626595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.626429 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9" event={"ID":"137c45cd-711f-46e6-a808-7e396102fb7f","Type":"ContainerDied","Data":"78e8764e71b7954af6bb89ddb473399a4444b6bb0a55e05aa8ec2d2fa45427f6"} Oct 30 14:46:42.626595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.626457 5824 scope.go:122] "RemoveContainer" containerID="ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f" Oct 30 14:46:42.637488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.637472 5824 scope.go:122] "RemoveContainer" containerID="ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f" Oct 30 14:46:42.637739 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:46:42.637720 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f\": container with ID starting with ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f not found: ID does not exist" containerID="ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f" Oct 30 14:46:42.637801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.637745 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f"} err="failed to get container status \"ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f\": rpc error: code = NotFound desc = could not find container \"ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f\": container with ID starting with ff5c875b20f9c43bd725ea1ef0365c376d098fe1c20fa370e33dcf248742218f not found: ID does not exist" Oct 30 14:46:42.686926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.686905 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9"] Oct 30 14:46:42.700449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:42.700423 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a7bb0c-5f9d6779f-jxjg9"] Oct 30 14:46:44.317797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:44.317763 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137c45cd-711f-46e6-a808-7e396102fb7f" path="/var/lib/kubelet/pods/137c45cd-711f-46e6-a808-7e396102fb7f/volumes" Oct 30 14:46:47.537096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.537060 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx"] Oct 30 14:46:47.538157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.538127 5824 cpu_manager.go:414] "containerMap: removing container" podUID="137c45cd-711f-46e6-a808-7e396102fb7f" containerName="payload-pre-processing" Oct 30 14:46:47.538157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.538155 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137c45cd-711f-46e6-a808-7e396102fb7f" containerName="payload-pre-processing" Oct 30 14:46:47.538412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.538245 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5cef0a4c-3529-4476-8ddf-f4e60e3f017e" containerName="revoke-keys" Oct 30 14:46:47.538412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.538255 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cef0a4c-3529-4476-8ddf-f4e60e3f017e" containerName="revoke-keys" Oct 30 14:46:47.538412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.538408 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="137c45cd-711f-46e6-a808-7e396102fb7f" containerName="payload-pre-processing" Oct 30 14:46:47.538572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.538425 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cef0a4c-3529-4476-8ddf-f4e60e3f017e" containerName="revoke-keys" Oct 30 14:46:47.555208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.545437 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.603958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.603929 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:46:47.604271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.604013 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:46:47.604400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.604380 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht"] Oct 30 14:46:47.608832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.608810 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx"] Oct 30 14:46:47.608931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.608913 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.613431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.613403 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht"] Oct 30 14:46:47.689972 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.689937 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nznbn\" (UniqueName: \"kubernetes.io/projected/f4cafb40-d317-4d4e-bf1d-b72650046fe1-kube-api-access-nznbn\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.690354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.690329 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.707956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.707909 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.708139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.707968 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.708139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.708037 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4pqx\" (UniqueName: \"kubernetes.io/projected/7534860f-19f0-4be5-8e90-a9b351094f54-kube-api-access-f4pqx\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.708139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.708128 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.821343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.821252 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4pqx\" (UniqueName: \"kubernetes.io/projected/7534860f-19f0-4be5-8e90-a9b351094f54-kube-api-access-f4pqx\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.826559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.826527 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.830842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.830710 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nznbn\" (UniqueName: \"kubernetes.io/projected/f4cafb40-d317-4d4e-bf1d-b72650046fe1-kube-api-access-nznbn\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.831525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.831502 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.835038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.835011 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.835695 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.835673 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.836007 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.830528 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.836234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.832504 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.838472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.838395 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.853214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.853081 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.857746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.857721 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4pqx\" (UniqueName: \"kubernetes.io/projected/7534860f-19f0-4be5-8e90-a9b351094f54-kube-api-access-f4pqx\") pod \"maas-api-revoke-sub-908d8733-gtbht\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:47.860398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.860371 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nznbn\" (UniqueName: \"kubernetes.io/projected/f4cafb40-d317-4d4e-bf1d-b72650046fe1-kube-api-access-nznbn\") pod \"maas-api-revoke-sub-9c5b2199-l9fvx\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.935699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.935674 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:47.945000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:47.944979 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:48.253615 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:48.253575 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cafb40_d317_4d4e_bf1d_b72650046fe1.slice/crio-cdde3be860be5f12c8202fb5ecfb2ec0c8583617e8885342915025d16f5456d2 WatchSource:0}: Error finding container cdde3be860be5f12c8202fb5ecfb2ec0c8583617e8885342915025d16f5456d2: Status 404 returned error can't find the container with id cdde3be860be5f12c8202fb5ecfb2ec0c8583617e8885342915025d16f5456d2 Oct 30 14:46:48.255169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.255025 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx"] Oct 30 14:46:48.272341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.272315 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht"] Oct 30 14:46:48.273011 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:48.272988 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7534860f_19f0_4be5_8e90_a9b351094f54.slice/crio-623b6e94f1c44677ee23926e197731716f6e193f56c0a0747ce56ebed5907355 WatchSource:0}: Error finding container 623b6e94f1c44677ee23926e197731716f6e193f56c0a0747ce56ebed5907355: Status 404 returned error can't find the container with id 623b6e94f1c44677ee23926e197731716f6e193f56c0a0747ce56ebed5907355 Oct 30 14:46:48.678637 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.678602 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" event={"ID":"f4cafb40-d317-4d4e-bf1d-b72650046fe1","Type":"ContainerStarted","Data":"df41910aba43f3c780ddd774197f93904291cf22b6e36b605b68ed48ff487dc4"} Oct 30 14:46:48.679051 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.678667 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" event={"ID":"f4cafb40-d317-4d4e-bf1d-b72650046fe1","Type":"ContainerStarted","Data":"cdde3be860be5f12c8202fb5ecfb2ec0c8583617e8885342915025d16f5456d2"} Oct 30 14:46:48.680334 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.680309 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" event={"ID":"7534860f-19f0-4be5-8e90-a9b351094f54","Type":"ContainerStarted","Data":"f010fed4ecd0250c9496df2b3601c384e45ab3fc1b623f5e2013bd307bb41fd1"} Oct 30 14:46:48.680463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.680357 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" event={"ID":"7534860f-19f0-4be5-8e90-a9b351094f54","Type":"ContainerStarted","Data":"623b6e94f1c44677ee23926e197731716f6e193f56c0a0747ce56ebed5907355"} Oct 30 14:46:48.716447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.716420 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr"] Oct 30 14:46:48.721110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.721086 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-396d70-qcvpr"] Oct 30 14:46:48.728792 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.728751 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" podStartSLOduration=1.728737074 podStartE2EDuration="1.728737074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:48.726970453 +0000 UTC m=+3438.527475898" watchObservedRunningTime="2026-10-30 14:46:48.728737074 +0000 UTC m=+3438.529242568" Oct 30 14:46:48.773951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:48.773894 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" podStartSLOduration=1.77388003 podStartE2EDuration="1.77388003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:48.772788617 +0000 UTC m=+3438.573294062" watchObservedRunningTime="2026-10-30 14:46:48.77388003 +0000 UTC m=+3438.574385503" Oct 30 14:46:49.048984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.048943 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc"] Oct 30 14:46:49.053533 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.053512 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.071469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.071445 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc"] Oct 30 14:46:49.154916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.154874 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.155090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.154930 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.155090 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.155043 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqwtv\" (UniqueName: \"kubernetes.io/projected/5f031b94-de5d-4983-b468-bad69a5b4b20-kube-api-access-mqwtv\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.256386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.256350 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.256547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.256401 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.256547 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.256439 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqwtv\" (UniqueName: \"kubernetes.io/projected/5f031b94-de5d-4983-b468-bad69a5b4b20-kube-api-access-mqwtv\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.257070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.257037 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.258942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.258918 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.282525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.282505 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqwtv\" (UniqueName: \"kubernetes.io/projected/5f031b94-de5d-4983-b468-bad69a5b4b20-kube-api-access-mqwtv\") pod \"maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.368869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.368847 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:49.683356 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:49.683320 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f031b94_de5d_4983_b468_bad69a5b4b20.slice/crio-778fb4bd79050a6973fea769a73737982bb06ae6774bc0297d4aa56c59c623be WatchSource:0}: Error finding container 778fb4bd79050a6973fea769a73737982bb06ae6774bc0297d4aa56c59c623be: Status 404 returned error can't find the container with id 778fb4bd79050a6973fea769a73737982bb06ae6774bc0297d4aa56c59c623be Oct 30 14:46:49.685153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.685127 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc"] Oct 30 14:46:49.689107 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.689079 5824 generic.go:356] "Generic (PLEG): container finished" podID="f4cafb40-d317-4d4e-bf1d-b72650046fe1" containerID="df41910aba43f3c780ddd774197f93904291cf22b6e36b605b68ed48ff487dc4" exitCode=0 Oct 30 14:46:49.689252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.689115 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" event={"ID":"f4cafb40-d317-4d4e-bf1d-b72650046fe1","Type":"ContainerDied","Data":"df41910aba43f3c780ddd774197f93904291cf22b6e36b605b68ed48ff487dc4"} Oct 30 14:46:49.690893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.690868 5824 generic.go:356] "Generic (PLEG): container finished" podID="7534860f-19f0-4be5-8e90-a9b351094f54" containerID="f010fed4ecd0250c9496df2b3601c384e45ab3fc1b623f5e2013bd307bb41fd1" exitCode=0 Oct 30 14:46:49.691004 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:49.690939 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" event={"ID":"7534860f-19f0-4be5-8e90-a9b351094f54","Type":"ContainerDied","Data":"f010fed4ecd0250c9496df2b3601c384e45ab3fc1b623f5e2013bd307bb41fd1"} Oct 30 14:46:50.321856 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.321786 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cef0a4c-3529-4476-8ddf-f4e60e3f017e" path="/var/lib/kubelet/pods/5cef0a4c-3529-4476-8ddf-f4e60e3f017e/volumes" Oct 30 14:46:50.704502 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.704459 5824 generic.go:356] "Generic (PLEG): container finished" podID="5f031b94-de5d-4983-b468-bad69a5b4b20" containerID="f90c9a8e7bee8b26433151e6fdf4b4f4521873a553c12fbb5ff347c04881c297" exitCode=0 Oct 30 14:46:50.704894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.704539 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" event={"ID":"5f031b94-de5d-4983-b468-bad69a5b4b20","Type":"ContainerDied","Data":"f90c9a8e7bee8b26433151e6fdf4b4f4521873a553c12fbb5ff347c04881c297"} Oct 30 14:46:50.704894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.704605 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" event={"ID":"5f031b94-de5d-4983-b468-bad69a5b4b20","Type":"ContainerStarted","Data":"778fb4bd79050a6973fea769a73737982bb06ae6774bc0297d4aa56c59c623be"} Oct 30 14:46:50.947805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.947782 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:50.951636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.951612 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:50.987137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987102 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-openshift-ca\") pod \"7534860f-19f0-4be5-8e90-a9b351094f54\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " Oct 30 14:46:50.987305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987145 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-openshift-ca\") pod \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " Oct 30 14:46:50.987305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987272 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-xks-ca\") pod \"7534860f-19f0-4be5-8e90-a9b351094f54\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " Oct 30 14:46:50.987405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987326 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cafb40-d317-4d4e-bf1d-b72650046fe1-kube-api-access-nznbn\" (UniqueName: \"kubernetes.io/projected/f4cafb40-d317-4d4e-bf1d-b72650046fe1-kube-api-access-nznbn\") pod \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " Oct 30 14:46:50.987405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987367 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-xks-ca\") pod \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\" (UID: \"f4cafb40-d317-4d4e-bf1d-b72650046fe1\") " Oct 30 14:46:50.987480 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987430 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7534860f-19f0-4be5-8e90-a9b351094f54-kube-api-access-f4pqx\" (UniqueName: \"kubernetes.io/projected/7534860f-19f0-4be5-8e90-a9b351094f54-kube-api-access-f4pqx\") pod \"7534860f-19f0-4be5-8e90-a9b351094f54\" (UID: \"7534860f-19f0-4be5-8e90-a9b351094f54\") " Oct 30 14:46:50.987514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987474 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-openshift-ca" pod "7534860f-19f0-4be5-8e90-a9b351094f54" (UID: "7534860f-19f0-4be5-8e90-a9b351094f54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:50.987582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987558 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-openshift-ca" pod "f4cafb40-d317-4d4e-bf1d-b72650046fe1" (UID: "f4cafb40-d317-4d4e-bf1d-b72650046fe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:50.987945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987910 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:50.988049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.987949 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:50.989542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.989516 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-xks-ca" pod "7534860f-19f0-4be5-8e90-a9b351094f54" (UID: "7534860f-19f0-4be5-8e90-a9b351094f54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:50.989542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.989526 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cafb40-d317-4d4e-bf1d-b72650046fe1-kube-api-access-nznbn" pod "f4cafb40-d317-4d4e-bf1d-b72650046fe1" (UID: "f4cafb40-d317-4d4e-bf1d-b72650046fe1"). InnerVolumeSpecName "kube-api-access-nznbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:50.989915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.989890 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-xks-ca" pod "f4cafb40-d317-4d4e-bf1d-b72650046fe1" (UID: "f4cafb40-d317-4d4e-bf1d-b72650046fe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:50.990014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:50.989914 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7534860f-19f0-4be5-8e90-a9b351094f54-kube-api-access-f4pqx" pod "7534860f-19f0-4be5-8e90-a9b351094f54" (UID: "7534860f-19f0-4be5-8e90-a9b351094f54"). InnerVolumeSpecName "kube-api-access-f4pqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:51.089227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.089174 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7534860f-19f0-4be5-8e90-a9b351094f54-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.089227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.089224 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nznbn\" (UniqueName: \"kubernetes.io/projected/f4cafb40-d317-4d4e-bf1d-b72650046fe1-kube-api-access-nznbn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.089427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.089238 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4cafb40-d317-4d4e-bf1d-b72650046fe1-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.089427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.089250 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4pqx\" (UniqueName: \"kubernetes.io/projected/7534860f-19f0-4be5-8e90-a9b351094f54-kube-api-access-f4pqx\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.362059 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv"] Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.363205 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7534860f-19f0-4be5-8e90-a9b351094f54" containerName="revoke-keys" Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.363225 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7534860f-19f0-4be5-8e90-a9b351094f54" containerName="revoke-keys" Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.363264 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f4cafb40-d317-4d4e-bf1d-b72650046fe1" containerName="revoke-keys" Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.363271 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cafb40-d317-4d4e-bf1d-b72650046fe1" containerName="revoke-keys" Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.363483 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7534860f-19f0-4be5-8e90-a9b351094f54" containerName="revoke-keys" Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.363502 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cafb40-d317-4d4e-bf1d-b72650046fe1" containerName="revoke-keys" Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.368184 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv"] Oct 30 14:46:51.370212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.368330 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.398154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.398119 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrs6s\" (UniqueName: \"kubernetes.io/projected/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-kube-api-access-nrs6s\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.398340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.398250 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.398454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.398436 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.501778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.501744 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.501914 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.501842 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrs6s\" (UniqueName: \"kubernetes.io/projected/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-kube-api-access-nrs6s\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.501914 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.501881 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.502620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.502598 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.504110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.504089 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.530415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.530390 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrs6s\" (UniqueName: \"kubernetes.io/projected/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-kube-api-access-nrs6s\") pod \"maas-api-revoke-sub-dcacc078-8z4sv\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.695340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.695320 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:51.702651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.702627 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c"] Oct 30 14:46:51.707840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.707820 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-33c209-gtz5c"] Oct 30 14:46:51.715942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.715921 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" Oct 30 14:46:51.716063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.715953 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx" event={"ID":"f4cafb40-d317-4d4e-bf1d-b72650046fe1","Type":"ContainerDied","Data":"cdde3be860be5f12c8202fb5ecfb2ec0c8583617e8885342915025d16f5456d2"} Oct 30 14:46:51.716063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.715997 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdde3be860be5f12c8202fb5ecfb2ec0c8583617e8885342915025d16f5456d2" Oct 30 14:46:51.717532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.717514 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" Oct 30 14:46:51.717641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.717549 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht" event={"ID":"7534860f-19f0-4be5-8e90-a9b351094f54","Type":"ContainerDied","Data":"623b6e94f1c44677ee23926e197731716f6e193f56c0a0747ce56ebed5907355"} Oct 30 14:46:51.717641 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.717628 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="623b6e94f1c44677ee23926e197731716f6e193f56c0a0747ce56ebed5907355" Oct 30 14:46:51.943723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:51.943697 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:52.008740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.007164 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-openshift-ca\") pod \"5f031b94-de5d-4983-b468-bad69a5b4b20\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " Oct 30 14:46:52.008740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.007287 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-xks-ca\") pod \"5f031b94-de5d-4983-b468-bad69a5b4b20\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " Oct 30 14:46:52.008740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.007821 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-openshift-ca" pod "5f031b94-de5d-4983-b468-bad69a5b4b20" (UID: "5f031b94-de5d-4983-b468-bad69a5b4b20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:52.008740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.008731 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f031b94-de5d-4983-b468-bad69a5b4b20-kube-api-access-mqwtv\" (UniqueName: \"kubernetes.io/projected/5f031b94-de5d-4983-b468-bad69a5b4b20-kube-api-access-mqwtv\") pod \"5f031b94-de5d-4983-b468-bad69a5b4b20\" (UID: \"5f031b94-de5d-4983-b468-bad69a5b4b20\") " Oct 30 14:46:52.009592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.009566 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:52.011336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.011307 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-xks-ca" pod "5f031b94-de5d-4983-b468-bad69a5b4b20" (UID: "5f031b94-de5d-4983-b468-bad69a5b4b20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:52.011783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.011748 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f031b94-de5d-4983-b468-bad69a5b4b20-kube-api-access-mqwtv" pod "5f031b94-de5d-4983-b468-bad69a5b4b20" (UID: "5f031b94-de5d-4983-b468-bad69a5b4b20"). InnerVolumeSpecName "kube-api-access-mqwtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:52.036518 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:52.036477 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd56fe2_85ca_473d_8db8_3ed75acd5c65.slice/crio-900538ce2b6276d5110af099d44e40a3e17926810f74de408e5eab6035fb3a36 WatchSource:0}: Error finding container 900538ce2b6276d5110af099d44e40a3e17926810f74de408e5eab6035fb3a36: Status 404 returned error can't find the container with id 900538ce2b6276d5110af099d44e40a3e17926810f74de408e5eab6035fb3a36 Oct 30 14:46:52.038145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.038108 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv"] Oct 30 14:46:52.110678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.110643 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqwtv\" (UniqueName: \"kubernetes.io/projected/5f031b94-de5d-4983-b468-bad69a5b4b20-kube-api-access-mqwtv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:52.110678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.110677 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f031b94-de5d-4983-b468-bad69a5b4b20-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:52.318238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.318121 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40aec71-9f6a-4bf0-b208-293e8c143308" path="/var/lib/kubelet/pods/c40aec71-9f6a-4bf0-b208-293e8c143308/volumes" Oct 30 14:46:52.726375 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.726350 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" Oct 30 14:46:52.726812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.726380 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc" event={"ID":"5f031b94-de5d-4983-b468-bad69a5b4b20","Type":"ContainerDied","Data":"778fb4bd79050a6973fea769a73737982bb06ae6774bc0297d4aa56c59c623be"} Oct 30 14:46:52.726812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.726432 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="778fb4bd79050a6973fea769a73737982bb06ae6774bc0297d4aa56c59c623be" Oct 30 14:46:52.728267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.728236 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" event={"ID":"7bd56fe2-85ca-473d-8db8-3ed75acd5c65","Type":"ContainerStarted","Data":"9fc509896e8f6e3778aa4400c07fa96b098655fc0e9a5b550e31fbf0448bdf30"} Oct 30 14:46:52.728380 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.728282 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" event={"ID":"7bd56fe2-85ca-473d-8db8-3ed75acd5c65","Type":"ContainerStarted","Data":"900538ce2b6276d5110af099d44e40a3e17926810f74de408e5eab6035fb3a36"} Oct 30 14:46:52.777170 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:52.777120 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" podStartSLOduration=1.7771068639999998 podStartE2EDuration="1.777106864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:46:52.772636465 +0000 UTC m=+3442.573141908" watchObservedRunningTime="2026-10-30 14:46:52.777106864 +0000 UTC m=+3442.577612306" Oct 30 14:46:53.737976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:53.737942 5824 generic.go:356] "Generic (PLEG): container finished" podID="7bd56fe2-85ca-473d-8db8-3ed75acd5c65" containerID="9fc509896e8f6e3778aa4400c07fa96b098655fc0e9a5b550e31fbf0448bdf30" exitCode=0 Oct 30 14:46:53.738471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:53.738026 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" event={"ID":"7bd56fe2-85ca-473d-8db8-3ed75acd5c65","Type":"ContainerDied","Data":"9fc509896e8f6e3778aa4400c07fa96b098655fc0e9a5b550e31fbf0448bdf30"} Oct 30 14:46:54.891891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:54.891862 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:54.947327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:54.947301 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-openshift-ca\") pod \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " Oct 30 14:46:54.947447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:54.947364 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-kube-api-access-nrs6s\" (UniqueName: \"kubernetes.io/projected/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-kube-api-access-nrs6s\") pod \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " Oct 30 14:46:54.947501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:54.947478 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-xks-ca\") pod \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\" (UID: \"7bd56fe2-85ca-473d-8db8-3ed75acd5c65\") " Oct 30 14:46:54.947719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:54.947690 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-openshift-ca" pod "7bd56fe2-85ca-473d-8db8-3ed75acd5c65" (UID: "7bd56fe2-85ca-473d-8db8-3ed75acd5c65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:46:54.947901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:54.947883 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:54.949423 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:54.949397 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-kube-api-access-nrs6s" pod "7bd56fe2-85ca-473d-8db8-3ed75acd5c65" (UID: "7bd56fe2-85ca-473d-8db8-3ed75acd5c65"). InnerVolumeSpecName "kube-api-access-nrs6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:46:54.949516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:54.949416 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-xks-ca" pod "7bd56fe2-85ca-473d-8db8-3ed75acd5c65" (UID: "7bd56fe2-85ca-473d-8db8-3ed75acd5c65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:46:55.049235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:55.049155 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:55.049235 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:55.049178 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrs6s\" (UniqueName: \"kubernetes.io/projected/7bd56fe2-85ca-473d-8db8-3ed75acd5c65-kube-api-access-nrs6s\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:46:55.758256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:55.758182 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" event={"ID":"7bd56fe2-85ca-473d-8db8-3ed75acd5c65","Type":"ContainerDied","Data":"900538ce2b6276d5110af099d44e40a3e17926810f74de408e5eab6035fb3a36"} Oct 30 14:46:55.758487 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:55.758276 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="900538ce2b6276d5110af099d44e40a3e17926810f74de408e5eab6035fb3a36" Oct 30 14:46:55.758487 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:55.758229 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv" Oct 30 14:46:57.998747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:57.998717 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw"] Oct 30 14:46:57.999449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:57.999433 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5f031b94-de5d-4983-b468-bad69a5b4b20" containerName="revoke-keys" Oct 30 14:46:57.999516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:57.999451 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f031b94-de5d-4983-b468-bad69a5b4b20" containerName="revoke-keys" Oct 30 14:46:57.999516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:57.999475 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7bd56fe2-85ca-473d-8db8-3ed75acd5c65" containerName="revoke-keys" Oct 30 14:46:57.999516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:57.999480 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd56fe2-85ca-473d-8db8-3ed75acd5c65" containerName="revoke-keys" Oct 30 14:46:57.999630 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:57.999575 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd56fe2-85ca-473d-8db8-3ed75acd5c65" containerName="revoke-keys" Oct 30 14:46:57.999630 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:57.999594 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f031b94-de5d-4983-b468-bad69a5b4b20" containerName="revoke-keys" Oct 30 14:46:58.003612 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.003593 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.026050 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.026015 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw"] Oct 30 14:46:58.083444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.083406 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thlz4\" (UniqueName: \"kubernetes.io/projected/f099f8b5-aa6a-454e-8c28-26c480082b17-kube-api-access-thlz4\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.083595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.083468 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.083595 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.083539 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.111581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.111558 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:46:58.111720 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.111704 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:46:58.185149 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.185123 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thlz4\" (UniqueName: \"kubernetes.io/projected/f099f8b5-aa6a-454e-8c28-26c480082b17-kube-api-access-thlz4\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.185284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.185162 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.185284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.185201 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.185730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.185709 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.187419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.187399 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.212539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.212517 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thlz4\" (UniqueName: \"kubernetes.io/projected/f099f8b5-aa6a-454e-8c28-26c480082b17-kube-api-access-thlz4\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.317627 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.317564 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:46:58.626392 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:46:58.626356 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf099f8b5_aa6a_454e_8c28_26c480082b17.slice/crio-09e983d113894bdf6ff4b7f95b79b99057e0a676d545ff336e6317127c246089 WatchSource:0}: Error finding container 09e983d113894bdf6ff4b7f95b79b99057e0a676d545ff336e6317127c246089: Status 404 returned error can't find the container with id 09e983d113894bdf6ff4b7f95b79b99057e0a676d545ff336e6317127c246089 Oct 30 14:46:58.627287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.627260 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw"] Oct 30 14:46:58.783321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:58.783285 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" event={"ID":"f099f8b5-aa6a-454e-8c28-26c480082b17","Type":"ContainerStarted","Data":"09e983d113894bdf6ff4b7f95b79b99057e0a676d545ff336e6317127c246089"} Oct 30 14:46:59.792135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:59.792060 5824 generic.go:356] "Generic (PLEG): container finished" podID="f099f8b5-aa6a-454e-8c28-26c480082b17" containerID="a5304bd69f25b67bd17cee45f1dab8de0ed1a843d28203efe424ef868b843db7" exitCode=0 Oct 30 14:46:59.792526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:46:59.792147 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" event={"ID":"f099f8b5-aa6a-454e-8c28-26c480082b17","Type":"ContainerDied","Data":"a5304bd69f25b67bd17cee45f1dab8de0ed1a843d28203efe424ef868b843db7"} Oct 30 14:47:00.475237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:00.474758 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m"] Oct 30 14:47:00.475237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:00.475009 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" containerID="cri-o://b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2" gracePeriod=30 Oct 30 14:47:00.964100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:00.964074 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:47:01.027101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.027074 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f099f8b5-aa6a-454e-8c28-26c480082b17-kube-api-access-thlz4\" (UniqueName: \"kubernetes.io/projected/f099f8b5-aa6a-454e-8c28-26c480082b17-kube-api-access-thlz4\") pod \"f099f8b5-aa6a-454e-8c28-26c480082b17\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " Oct 30 14:47:01.027326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.027269 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-xks-ca\") pod \"f099f8b5-aa6a-454e-8c28-26c480082b17\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " Oct 30 14:47:01.027326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.027310 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-openshift-ca\") pod \"f099f8b5-aa6a-454e-8c28-26c480082b17\" (UID: \"f099f8b5-aa6a-454e-8c28-26c480082b17\") " Oct 30 14:47:01.027958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.027932 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-openshift-ca" pod "f099f8b5-aa6a-454e-8c28-26c480082b17" (UID: "f099f8b5-aa6a-454e-8c28-26c480082b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:01.029992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.029965 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-xks-ca" pod "f099f8b5-aa6a-454e-8c28-26c480082b17" (UID: "f099f8b5-aa6a-454e-8c28-26c480082b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:01.031362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.031335 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f099f8b5-aa6a-454e-8c28-26c480082b17-kube-api-access-thlz4" pod "f099f8b5-aa6a-454e-8c28-26c480082b17" (UID: "f099f8b5-aa6a-454e-8c28-26c480082b17"). InnerVolumeSpecName "kube-api-access-thlz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:01.129688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.129607 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:01.129688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.129636 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f099f8b5-aa6a-454e-8c28-26c480082b17-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:01.129688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.129650 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thlz4\" (UniqueName: \"kubernetes.io/projected/f099f8b5-aa6a-454e-8c28-26c480082b17-kube-api-access-thlz4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:01.808319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.808292 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" Oct 30 14:47:01.808471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.808337 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw" event={"ID":"f099f8b5-aa6a-454e-8c28-26c480082b17","Type":"ContainerDied","Data":"09e983d113894bdf6ff4b7f95b79b99057e0a676d545ff336e6317127c246089"} Oct 30 14:47:01.808471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:01.808386 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e983d113894bdf6ff4b7f95b79b99057e0a676d545ff336e6317127c246089" Oct 30 14:47:03.640277 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:47:03.640237 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:47:05.762586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.762528 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:47:05.839560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.839528 5824 generic.go:356] "Generic (PLEG): container finished" podID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerID="b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2" exitCode=0 Oct 30 14:47:05.839825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.839611 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" Oct 30 14:47:05.839959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.839613 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" event={"ID":"06f88aaf-4caa-47d3-badd-9a26b264052b","Type":"ContainerDied","Data":"b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2"} Oct 30 14:47:05.840086 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.840069 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m" event={"ID":"06f88aaf-4caa-47d3-badd-9a26b264052b","Type":"ContainerDied","Data":"5d9556474ee09bcee6fe6fe066f0ec018cbcd70515283e128514548cec37b918"} Oct 30 14:47:05.840179 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.840106 5824 scope.go:122] "RemoveContainer" containerID="b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2" Oct 30 14:47:05.853112 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.853091 5824 scope.go:122] "RemoveContainer" containerID="a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c" Oct 30 14:47:05.863074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.863047 5824 scope.go:122] "RemoveContainer" containerID="b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2" Oct 30 14:47:05.863363 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:47:05.863337 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2\": container with ID starting with b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2 not found: ID does not exist" containerID="b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2" Oct 30 14:47:05.863468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.863371 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2"} err="failed to get container status \"b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2\": rpc error: code = NotFound desc = could not find container \"b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2\": container with ID starting with b77ce965dc32c10baa067ba40db2e4f8ab2ff4b2047b63a64e4e15686560f1b2 not found: ID does not exist" Oct 30 14:47:05.863468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.863392 5824 scope.go:122] "RemoveContainer" containerID="a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c" Oct 30 14:47:05.863649 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:47:05.863631 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c\": container with ID starting with a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c not found: ID does not exist" containerID="a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c" Oct 30 14:47:05.863713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.863656 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c"} err="failed to get container status \"a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c\": rpc error: code = NotFound desc = could not find container \"a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c\": container with ID starting with a6862e931b730e0e30347ad028a5b2152424902e2d95188598337122dc53b57c not found: ID does not exist" Oct 30 14:47:05.899504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899470 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-podinfo\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.899639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899532 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-socket\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.899710 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899645 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-data\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.899776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899700 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-token\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-token\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.899833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899783 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-envoy\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.899833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899811 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-certs\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.899950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899818 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-socket" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:05.899950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899845 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-credential-socket\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.899950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899932 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-data" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:05.900122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.899987 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f88aaf-4caa-47d3-badd-9a26b264052b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f88aaf-4caa-47d3-badd-9a26b264052b-istiod-ca-cert\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.900122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900041 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-credential-socket" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:05.900122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900080 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-kube-api-access-8npk5\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-kube-api-access-8npk5\") pod \"06f88aaf-4caa-47d3-badd-9a26b264052b\" (UID: \"06f88aaf-4caa-47d3-badd-9a26b264052b\") " Oct 30 14:47:05.901023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900125 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-certs" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:05.901023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900426 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f88aaf-4caa-47d3-badd-9a26b264052b-istiod-ca-cert" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:05.901023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900777 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:05.901023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900800 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:05.901023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900815 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:05.901023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900830 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f88aaf-4caa-47d3-badd-9a26b264052b-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:05.901023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.900844 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:05.901865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.901840 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-podinfo" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:47:05.902199 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.902165 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-token" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:05.902321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.902298 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-kube-api-access-8npk5" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "kube-api-access-8npk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:05.902454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:05.902435 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-envoy" pod "06f88aaf-4caa-47d3-badd-9a26b264052b" (UID: "06f88aaf-4caa-47d3-badd-9a26b264052b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:47:06.001859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:06.001834 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:06.001859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:06.001859 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:06.002023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:06.001869 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8npk5\" (UniqueName: \"kubernetes.io/projected/06f88aaf-4caa-47d3-badd-9a26b264052b-kube-api-access-8npk5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:06.002023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:06.001879 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f88aaf-4caa-47d3-badd-9a26b264052b-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:06.212439 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:06.212406 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m"] Oct 30 14:47:06.224984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:06.224955 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-33c209-openshift-default-8dc5b9dc-gfl9m"] Oct 30 14:47:06.377233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:06.375824 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" path="/var/lib/kubelet/pods/06f88aaf-4caa-47d3-badd-9a26b264052b/volumes" Oct 30 14:47:11.044299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:11.044257 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn"] Oct 30 14:47:11.061762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:11.059656 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3003b5-jt2pn"] Oct 30 14:47:12.317815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:12.317781 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d817e56-c250-4ecc-9865-b2c7e2a0d72d" path="/var/lib/kubelet/pods/5d817e56-c250-4ecc-9865-b2c7e2a0d72d/volumes" Oct 30 14:47:15.932309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.932273 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7"] Oct 30 14:47:15.932936 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.932921 5824 cpu_manager.go:414] "containerMap: removing container" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" Oct 30 14:47:15.932984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.932938 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" Oct 30 14:47:15.932984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.932948 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f099f8b5-aa6a-454e-8c28-26c480082b17" containerName="revoke-keys" Oct 30 14:47:15.932984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.932953 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f099f8b5-aa6a-454e-8c28-26c480082b17" containerName="revoke-keys" Oct 30 14:47:15.932984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.932983 5824 cpu_manager.go:414] "containerMap: removing container" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" Oct 30 14:47:15.933142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.932989 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" Oct 30 14:47:15.933142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.933105 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" Oct 30 14:47:15.933142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.933121 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f099f8b5-aa6a-454e-8c28-26c480082b17" containerName="revoke-keys" Oct 30 14:47:15.938291 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.938268 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:15.954781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.954754 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7"] Oct 30 14:47:15.964095 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.964070 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:47:15.964325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:15.964304 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:47:16.041357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.041322 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.041508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.041495 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.041580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.041561 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkpr\" (UniqueName: \"kubernetes.io/projected/5267e476-7724-4f1c-ac3c-c18cb79138a2-kube-api-access-5nkpr\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.142811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.142786 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.142960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.142937 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.143044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.143025 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkpr\" (UniqueName: \"kubernetes.io/projected/5267e476-7724-4f1c-ac3c-c18cb79138a2-kube-api-access-5nkpr\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.143428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.143409 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.145258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.145238 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.169747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.169721 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkpr\" (UniqueName: \"kubernetes.io/projected/5267e476-7724-4f1c-ac3c-c18cb79138a2-kube-api-access-5nkpr\") pod \"maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.252236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.252134 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:16.560804 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:47:16.560775 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5267e476_7724_4f1c_ac3c_c18cb79138a2.slice/crio-08608f8ec45e1b7d91fc88d3dc8906a7f9def849db8b103de92be87903bcc232 WatchSource:0}: Error finding container 08608f8ec45e1b7d91fc88d3dc8906a7f9def849db8b103de92be87903bcc232: Status 404 returned error can't find the container with id 08608f8ec45e1b7d91fc88d3dc8906a7f9def849db8b103de92be87903bcc232 Oct 30 14:47:16.562081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.562025 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7"] Oct 30 14:47:16.920246 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.920211 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" event={"ID":"5267e476-7724-4f1c-ac3c-c18cb79138a2","Type":"ContainerStarted","Data":"b59400a8214552b1747c1a21ae930441ec8279b2e0f62b7dec365223dc6050a0"} Oct 30 14:47:16.920385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.920273 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" event={"ID":"5267e476-7724-4f1c-ac3c-c18cb79138a2","Type":"ContainerStarted","Data":"08608f8ec45e1b7d91fc88d3dc8906a7f9def849db8b103de92be87903bcc232"} Oct 30 14:47:16.967344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:16.967295 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" podStartSLOduration=1.9672806120000002 podStartE2EDuration="1.967280612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:16.964457277 +0000 UTC m=+3466.764962720" watchObservedRunningTime="2026-10-30 14:47:16.967280612 +0000 UTC m=+3466.767786053" Oct 30 14:47:17.929954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:17.929920 5824 generic.go:356] "Generic (PLEG): container finished" podID="5267e476-7724-4f1c-ac3c-c18cb79138a2" containerID="b59400a8214552b1747c1a21ae930441ec8279b2e0f62b7dec365223dc6050a0" exitCode=0 Oct 30 14:47:17.930113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:17.930001 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" event={"ID":"5267e476-7724-4f1c-ac3c-c18cb79138a2","Type":"ContainerDied","Data":"b59400a8214552b1747c1a21ae930441ec8279b2e0f62b7dec365223dc6050a0"} Oct 30 14:47:19.081019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.080997 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:19.179944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.179914 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-xks-ca\") pod \"5267e476-7724-4f1c-ac3c-c18cb79138a2\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " Oct 30 14:47:19.180091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.179959 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-openshift-ca\") pod \"5267e476-7724-4f1c-ac3c-c18cb79138a2\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " Oct 30 14:47:19.180091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.179986 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5267e476-7724-4f1c-ac3c-c18cb79138a2-kube-api-access-5nkpr\" (UniqueName: \"kubernetes.io/projected/5267e476-7724-4f1c-ac3c-c18cb79138a2-kube-api-access-5nkpr\") pod \"5267e476-7724-4f1c-ac3c-c18cb79138a2\" (UID: \"5267e476-7724-4f1c-ac3c-c18cb79138a2\") " Oct 30 14:47:19.180315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.180290 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-openshift-ca" pod "5267e476-7724-4f1c-ac3c-c18cb79138a2" (UID: "5267e476-7724-4f1c-ac3c-c18cb79138a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:19.180491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.180477 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:19.181945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.181921 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-xks-ca" pod "5267e476-7724-4f1c-ac3c-c18cb79138a2" (UID: "5267e476-7724-4f1c-ac3c-c18cb79138a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:19.182241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.182220 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5267e476-7724-4f1c-ac3c-c18cb79138a2-kube-api-access-5nkpr" pod "5267e476-7724-4f1c-ac3c-c18cb79138a2" (UID: "5267e476-7724-4f1c-ac3c-c18cb79138a2"). InnerVolumeSpecName "kube-api-access-5nkpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:19.281302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.281273 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5267e476-7724-4f1c-ac3c-c18cb79138a2-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:19.281302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.281299 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkpr\" (UniqueName: \"kubernetes.io/projected/5267e476-7724-4f1c-ac3c-c18cb79138a2-kube-api-access-5nkpr\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:19.945404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.945376 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" Oct 30 14:47:19.945589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.945406 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7" event={"ID":"5267e476-7724-4f1c-ac3c-c18cb79138a2","Type":"ContainerDied","Data":"08608f8ec45e1b7d91fc88d3dc8906a7f9def849db8b103de92be87903bcc232"} Oct 30 14:47:19.945589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:19.945458 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08608f8ec45e1b7d91fc88d3dc8906a7f9def849db8b103de92be87903bcc232" Oct 30 14:47:21.052297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:21.052263 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d"] Oct 30 14:47:21.069719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:21.069696 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd77c42d-7qz7d"] Oct 30 14:47:22.319985 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:22.319947 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f080aad9-b186-4a5e-be3a-597a046e836a" path="/var/lib/kubelet/pods/f080aad9-b186-4a5e-be3a-597a046e836a/volumes" Oct 30 14:47:28.205270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.205223 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7"] Oct 30 14:47:28.206129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.206107 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5267e476-7724-4f1c-ac3c-c18cb79138a2" containerName="revoke-keys" Oct 30 14:47:28.206282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.206130 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5267e476-7724-4f1c-ac3c-c18cb79138a2" containerName="revoke-keys" Oct 30 14:47:28.206398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.206382 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f88aaf-4caa-47d3-badd-9a26b264052b" containerName="istio-proxy" Oct 30 14:47:28.206460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.206407 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5267e476-7724-4f1c-ac3c-c18cb79138a2" containerName="revoke-keys" Oct 30 14:47:28.211996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.211974 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.234245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.234222 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:47:28.243006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.242985 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:47:28.247688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.247666 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7"] Oct 30 14:47:28.283910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.283882 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vm7j\" (UniqueName: \"kubernetes.io/projected/01929d1a-4988-4775-b40c-915239158e7e-kube-api-access-9vm7j\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.284047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.283926 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01929d1a-4988-4775-b40c-915239158e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.284143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.284124 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01929d1a-4988-4775-b40c-915239158e7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.385600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.385572 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vm7j\" (UniqueName: \"kubernetes.io/projected/01929d1a-4988-4775-b40c-915239158e7e-kube-api-access-9vm7j\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.385718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.385620 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01929d1a-4988-4775-b40c-915239158e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.385782 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.385767 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01929d1a-4988-4775-b40c-915239158e7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.386272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.386245 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01929d1a-4988-4775-b40c-915239158e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.388079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.388051 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01929d1a-4988-4775-b40c-915239158e7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.413810 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.413783 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vm7j\" (UniqueName: \"kubernetes.io/projected/01929d1a-4988-4775-b40c-915239158e7e-kube-api-access-9vm7j\") pod \"maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.526371 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.526304 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:28.832830 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:47:28.832804 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01929d1a_4988_4775_b40c_915239158e7e.slice/crio-25faa53cf1c38664e531d7dbd4f55ef26ce50e1149d6d844e859118d331b4739 WatchSource:0}: Error finding container 25faa53cf1c38664e531d7dbd4f55ef26ce50e1149d6d844e859118d331b4739: Status 404 returned error can't find the container with id 25faa53cf1c38664e531d7dbd4f55ef26ce50e1149d6d844e859118d331b4739 Oct 30 14:47:28.833037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:28.833016 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7"] Oct 30 14:47:29.013662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:29.013625 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" event={"ID":"01929d1a-4988-4775-b40c-915239158e7e","Type":"ContainerStarted","Data":"25faa53cf1c38664e531d7dbd4f55ef26ce50e1149d6d844e859118d331b4739"} Oct 30 14:47:30.022491 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:30.022404 5824 generic.go:356] "Generic (PLEG): container finished" podID="01929d1a-4988-4775-b40c-915239158e7e" containerID="45ec94fcc9ee51f27239b6470ffc451a0d3067c91d5a2a62e12e3c68ffc778c3" exitCode=0 Oct 30 14:47:30.022895 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:30.022483 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" event={"ID":"01929d1a-4988-4775-b40c-915239158e7e","Type":"ContainerDied","Data":"45ec94fcc9ee51f27239b6470ffc451a0d3067c91d5a2a62e12e3c68ffc778c3"} Oct 30 14:47:31.173721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.173697 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:31.320110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.320032 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01929d1a-4988-4775-b40c-915239158e7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01929d1a-4988-4775-b40c-915239158e7e-maas-api-openshift-ca\") pod \"01929d1a-4988-4775-b40c-915239158e7e\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " Oct 30 14:47:31.320269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.320136 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01929d1a-4988-4775-b40c-915239158e7e-kube-api-access-9vm7j\" (UniqueName: \"kubernetes.io/projected/01929d1a-4988-4775-b40c-915239158e7e-kube-api-access-9vm7j\") pod \"01929d1a-4988-4775-b40c-915239158e7e\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " Oct 30 14:47:31.320336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.320279 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01929d1a-4988-4775-b40c-915239158e7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01929d1a-4988-4775-b40c-915239158e7e-maas-api-xks-ca\") pod \"01929d1a-4988-4775-b40c-915239158e7e\" (UID: \"01929d1a-4988-4775-b40c-915239158e7e\") " Oct 30 14:47:31.320415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.320392 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01929d1a-4988-4775-b40c-915239158e7e-maas-api-openshift-ca" pod "01929d1a-4988-4775-b40c-915239158e7e" (UID: "01929d1a-4988-4775-b40c-915239158e7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:31.320749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.320730 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01929d1a-4988-4775-b40c-915239158e7e-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:31.322239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.322214 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01929d1a-4988-4775-b40c-915239158e7e-maas-api-xks-ca" pod "01929d1a-4988-4775-b40c-915239158e7e" (UID: "01929d1a-4988-4775-b40c-915239158e7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:31.322347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.322248 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01929d1a-4988-4775-b40c-915239158e7e-kube-api-access-9vm7j" pod "01929d1a-4988-4775-b40c-915239158e7e" (UID: "01929d1a-4988-4775-b40c-915239158e7e"). InnerVolumeSpecName "kube-api-access-9vm7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:31.422427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.422404 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vm7j\" (UniqueName: \"kubernetes.io/projected/01929d1a-4988-4775-b40c-915239158e7e-kube-api-access-9vm7j\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:31.422427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:31.422427 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01929d1a-4988-4775-b40c-915239158e7e-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:32.038306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:32.038276 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" Oct 30 14:47:32.038467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:32.038308 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7" event={"ID":"01929d1a-4988-4775-b40c-915239158e7e","Type":"ContainerDied","Data":"25faa53cf1c38664e531d7dbd4f55ef26ce50e1149d6d844e859118d331b4739"} Oct 30 14:47:32.038467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:32.038352 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25faa53cf1c38664e531d7dbd4f55ef26ce50e1149d6d844e859118d331b4739" Oct 30 14:47:33.643973 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:47:33.643938 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:47:37.373433 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.373389 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g"] Oct 30 14:47:37.373879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.373625 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" podUID="60c9278f-54d3-49cb-975c-f49372a9fd1c" containerName="maas-api" containerID="cri-o://e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25" gracePeriod=30 Oct 30 14:47:37.667686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.667320 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:47:37.800113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.800081 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c9278f-54d3-49cb-975c-f49372a9fd1c-kube-api-access-kjvr9\" (UniqueName: \"kubernetes.io/projected/60c9278f-54d3-49cb-975c-f49372a9fd1c-kube-api-access-kjvr9\") pod \"60c9278f-54d3-49cb-975c-f49372a9fd1c\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " Oct 30 14:47:37.800347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.800219 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c9278f-54d3-49cb-975c-f49372a9fd1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c9278f-54d3-49cb-975c-f49372a9fd1c-openshift-service-ca\") pod \"60c9278f-54d3-49cb-975c-f49372a9fd1c\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " Oct 30 14:47:37.800347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.800277 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-maas-api-tls\") pod \"60c9278f-54d3-49cb-975c-f49372a9fd1c\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " Oct 30 14:47:37.800347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.800325 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-metrics-certs\") pod \"60c9278f-54d3-49cb-975c-f49372a9fd1c\" (UID: \"60c9278f-54d3-49cb-975c-f49372a9fd1c\") " Oct 30 14:47:37.800670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.800635 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c9278f-54d3-49cb-975c-f49372a9fd1c-openshift-service-ca" pod "60c9278f-54d3-49cb-975c-f49372a9fd1c" (UID: "60c9278f-54d3-49cb-975c-f49372a9fd1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:47:37.800901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.800881 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c9278f-54d3-49cb-975c-f49372a9fd1c-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:37.802355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.802332 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c9278f-54d3-49cb-975c-f49372a9fd1c-kube-api-access-kjvr9" pod "60c9278f-54d3-49cb-975c-f49372a9fd1c" (UID: "60c9278f-54d3-49cb-975c-f49372a9fd1c"). InnerVolumeSpecName "kube-api-access-kjvr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:47:37.802355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.802343 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-maas-api-tls" pod "60c9278f-54d3-49cb-975c-f49372a9fd1c" (UID: "60c9278f-54d3-49cb-975c-f49372a9fd1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:37.802483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.802367 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-metrics-certs" pod "60c9278f-54d3-49cb-975c-f49372a9fd1c" (UID: "60c9278f-54d3-49cb-975c-f49372a9fd1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:47:37.913301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.906256 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:37.913301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.906290 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c9278f-54d3-49cb-975c-f49372a9fd1c-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:37.913301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:37.906304 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjvr9\" (UniqueName: \"kubernetes.io/projected/60c9278f-54d3-49cb-975c-f49372a9fd1c-kube-api-access-kjvr9\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:47:38.097136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.097099 5824 generic.go:356] "Generic (PLEG): container finished" podID="60c9278f-54d3-49cb-975c-f49372a9fd1c" containerID="e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25" exitCode=0 Oct 30 14:47:38.097349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.097154 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" Oct 30 14:47:38.097349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.097160 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" event={"ID":"60c9278f-54d3-49cb-975c-f49372a9fd1c","Type":"ContainerDied","Data":"e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25"} Oct 30 14:47:38.097349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.097235 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g" event={"ID":"60c9278f-54d3-49cb-975c-f49372a9fd1c","Type":"ContainerDied","Data":"81ea3e8c48b7d5aa456a17a9f0ec22abd2a6bd0b85e188b66a10f91ce7cbc53d"} Oct 30 14:47:38.097349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.097265 5824 scope.go:122] "RemoveContainer" containerID="e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25" Oct 30 14:47:38.111143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.111125 5824 scope.go:122] "RemoveContainer" containerID="e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25" Oct 30 14:47:38.111452 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:47:38.111432 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25\": container with ID starting with e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25 not found: ID does not exist" containerID="e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25" Oct 30 14:47:38.111538 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.111462 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25"} err="failed to get container status \"e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25\": rpc error: code = NotFound desc = could not find container \"e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25\": container with ID starting with e2b905e9e8a1ffd7d34e94ade5ff03a07a9222b8d55dcb2a9dee61acec2fab25 not found: ID does not exist" Oct 30 14:47:38.164450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.164394 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g"] Oct 30 14:47:38.178369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.178346 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-a7bb0c-6567cd768-xqf9g"] Oct 30 14:47:38.238038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.238004 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj"] Oct 30 14:47:38.241383 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.241353 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a7bb0c-gn4xj"] Oct 30 14:47:38.318528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.318493 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267196fa-7db1-4e88-9e2a-92dc46b83cd4" path="/var/lib/kubelet/pods/267196fa-7db1-4e88-9e2a-92dc46b83cd4/volumes" Oct 30 14:47:38.319002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:38.318982 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c9278f-54d3-49cb-975c-f49372a9fd1c" path="/var/lib/kubelet/pods/60c9278f-54d3-49cb-975c-f49372a9fd1c/volumes" Oct 30 14:47:41.696296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:41.696239 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt"] Oct 30 14:47:41.698436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:41.698411 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0e8b9-78llt"] Oct 30 14:47:42.317727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:42.317694 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecb7561-c998-4ecf-8f24-d697654a3161" path="/var/lib/kubelet/pods/1ecb7561-c998-4ecf-8f24-d697654a3161/volumes" Oct 30 14:47:44.186941 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:44.186902 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw"] Oct 30 14:47:44.200159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:44.200135 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e91df-gf6mw"] Oct 30 14:47:44.318845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:44.318816 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f099f8b5-aa6a-454e-8c28-26c480082b17" path="/var/lib/kubelet/pods/f099f8b5-aa6a-454e-8c28-26c480082b17/volumes" Oct 30 14:47:44.916925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:44.916888 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc"] Oct 30 14:47:44.926277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:44.926247 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6176c3-bshfc"] Oct 30 14:47:46.318071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:47:46.318037 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f031b94-de5d-4983-b468-bad69a5b4b20" path="/var/lib/kubelet/pods/5f031b94-de5d-4983-b468-bad69a5b4b20/volumes" Oct 30 14:48:03.336285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:03.336249 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4"] Oct 30 14:48:03.336835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:03.336457 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" containerID="cri-o://7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76" gracePeriod=30 Oct 30 14:48:03.640963 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:03.640876 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:48:08.602059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.602002 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:48:08.678369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.678337 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-kube-api-access-5jhhv\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-kube-api-access-5jhhv\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.678534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.678432 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-credential-socket\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.678534 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.678509 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-data\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.678671 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.678648 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-token\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-token\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.678730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.678719 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-podinfo\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.678858 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.678825 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-certs\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.680607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.679081 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-credential-socket" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:08.680607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.679628 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-data" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:08.680607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.679928 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-certs" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:08.681091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.680831 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-socket" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:08.681181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.680534 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-socket\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.682876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.682345 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce1b7ad-c03a-457d-9f05-100e28d5886e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce1b7ad-c03a-457d-9f05-100e28d5886e-istiod-ca-cert\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.683105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.683088 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-envoy\") pod \"bce1b7ad-c03a-457d-9f05-100e28d5886e\" (UID: \"bce1b7ad-c03a-457d-9f05-100e28d5886e\") " Oct 30 14:48:08.683351 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.683235 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-token" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:08.684148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.684121 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.684311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.684153 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.684311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.684167 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.684311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.684181 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.684311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.684208 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.685169 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.685142 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce1b7ad-c03a-457d-9f05-100e28d5886e-istiod-ca-cert" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:08.690222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.686377 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-podinfo" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:08.690222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.688568 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-envoy" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:08.690222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.688804 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-kube-api-access-5jhhv" pod "bce1b7ad-c03a-457d-9f05-100e28d5886e" (UID: "bce1b7ad-c03a-457d-9f05-100e28d5886e"). InnerVolumeSpecName "kube-api-access-5jhhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:08.785951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.785924 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce1b7ad-c03a-457d-9f05-100e28d5886e-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.785951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.785949 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.786139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.785959 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jhhv\" (UniqueName: \"kubernetes.io/projected/bce1b7ad-c03a-457d-9f05-100e28d5886e-kube-api-access-5jhhv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.786139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.785973 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce1b7ad-c03a-457d-9f05-100e28d5886e-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:08.942813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:08.941821 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc"] Oct 30 14:48:09.334067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.333981 5824 generic.go:356] "Generic (PLEG): container finished" podID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerID="7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76" exitCode=0 Oct 30 14:48:09.334067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.334057 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" Oct 30 14:48:09.334315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.334075 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" event={"ID":"bce1b7ad-c03a-457d-9f05-100e28d5886e","Type":"ContainerDied","Data":"7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76"} Oct 30 14:48:09.334315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.334131 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4" event={"ID":"bce1b7ad-c03a-457d-9f05-100e28d5886e","Type":"ContainerDied","Data":"a8fc84f82186ee4e5302e507e6f320b9058672e1072ff10e3a36365c43a0482b"} Oct 30 14:48:09.334315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.334164 5824 scope.go:122] "RemoveContainer" containerID="7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76" Oct 30 14:48:09.346100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.346081 5824 scope.go:122] "RemoveContainer" containerID="7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76" Oct 30 14:48:09.346383 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:09.346362 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76\": container with ID starting with 7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76 not found: ID does not exist" containerID="7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76" Oct 30 14:48:09.346467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.346388 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76"} err="failed to get container status \"7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76\": rpc error: code = NotFound desc = could not find container \"7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76\": container with ID starting with 7b37091a3fb22b78313b5dcc8825117d6ee076abbe3827268151be77a0ae3c76 not found: ID does not exist" Oct 30 14:48:09.395343 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.395312 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4"] Oct 30 14:48:09.407846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:09.407818 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6e91df-openshift-default-8689d4c967-k6jj4"] Oct 30 14:48:10.318627 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:10.318595 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" path="/var/lib/kubelet/pods/bce1b7ad-c03a-457d-9f05-100e28d5886e/volumes" Oct 30 14:48:12.399180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:12.399086 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7"] Oct 30 14:48:12.414318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:12.412552 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d05e21-rpbd7"] Oct 30 14:48:14.318977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:14.318947 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5267e476-7724-4f1c-ac3c-c18cb79138a2" path="/var/lib/kubelet/pods/5267e476-7724-4f1c-ac3c-c18cb79138a2/volumes" Oct 30 14:48:14.501625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:14.501596 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt"] Oct 30 14:48:14.538575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:14.538548 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j"] Oct 30 14:48:19.501897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:19.501861 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" podUID="79f5cdfb-5ad6-48a6-93c3-bbdeae520169" containerName="payload-processing" containerID="cri-o://a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77" gracePeriod=25 Oct 30 14:48:19.539590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:19.539559 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" podUID="09587b10-c0dd-49ca-be55-ba1ba847c2b9" containerName="payload-pre-processing" containerID="cri-o://df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc" gracePeriod=25 Oct 30 14:48:20.037661 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.037631 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d"] Oct 30 14:48:20.037851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.037825 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" podUID="3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" containerName="maas-api" containerID="cri-o://fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0" gracePeriod=30 Oct 30 14:48:20.322175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.322148 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:48:20.419775 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.419729 5824 generic.go:356] "Generic (PLEG): container finished" podID="3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" containerID="fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0" exitCode=0 Oct 30 14:48:20.419955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.419776 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" event={"ID":"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a","Type":"ContainerDied","Data":"fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0"} Oct 30 14:48:20.419955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.419859 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" event={"ID":"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a","Type":"ContainerDied","Data":"a7061ad4abe34bfd68aac97462baf35131ccfbb3ef22cb6194eee61e65e1c72b"} Oct 30 14:48:20.419955 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.419891 5824 scope.go:122] "RemoveContainer" containerID="fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0" Oct 30 14:48:20.420159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.420136 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d" Oct 30 14:48:20.431841 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.431818 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f"] Oct 30 14:48:20.432122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.432054 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" containerID="cri-o://9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723" gracePeriod=30 Oct 30 14:48:20.436437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.436417 5824 scope.go:122] "RemoveContainer" containerID="fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0" Oct 30 14:48:20.436939 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:20.436887 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0\": container with ID starting with fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0 not found: ID does not exist" containerID="fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0" Oct 30 14:48:20.436939 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.436928 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0"} err="failed to get container status \"fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0\": rpc error: code = NotFound desc = could not find container \"fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0\": container with ID starting with fb4ba460a1b41acb83ea698d1a032a50b900f4ab738c7e160f13c6c17c603bc0 not found: ID does not exist" Oct 30 14:48:20.439339 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.439317 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j"] Oct 30 14:48:20.439499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.439479 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" containerID="cri-o://0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d" gracePeriod=30 Oct 30 14:48:20.443237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.443169 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-maas-api-tls\") pod \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " Oct 30 14:48:20.443336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.443258 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-openshift-service-ca\") pod \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " Oct 30 14:48:20.443336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.443330 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-kube-api-access-w8dng\" (UniqueName: \"kubernetes.io/projected/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-kube-api-access-w8dng\") pod \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " Oct 30 14:48:20.443431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.443384 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-metrics-certs\") pod \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\" (UID: \"3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a\") " Oct 30 14:48:20.443639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.443614 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-openshift-service-ca" pod "3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" (UID: "3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:20.444332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.444308 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:20.445504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.445480 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-maas-api-tls" pod "3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" (UID: "3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:20.445909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.445889 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-metrics-certs" pod "3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" (UID: "3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:20.445989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.445924 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-kube-api-access-w8dng" pod "3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" (UID: "3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a"). InnerVolumeSpecName "kube-api-access-w8dng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:20.558271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.558137 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:20.558271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.558167 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8dng\" (UniqueName: \"kubernetes.io/projected/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-kube-api-access-w8dng\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:20.558271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.558177 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:20.780931 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.780897 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d"] Oct 30 14:48:20.786904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:20.786873 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2927fd-5cb6b8c87c-dn96d"] Oct 30 14:48:21.010126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.010079 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7"] Oct 30 14:48:21.022857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.022788 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2927fd-nwjw7"] Oct 30 14:48:21.083504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.083469 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf"] Oct 30 14:48:21.084116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084100 5824 cpu_manager.go:414] "containerMap: removing container" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" Oct 30 14:48:21.084116 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084115 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" Oct 30 14:48:21.084303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084136 5824 cpu_manager.go:414] "containerMap: removing container" podUID="01929d1a-4988-4775-b40c-915239158e7e" containerName="revoke-keys" Oct 30 14:48:21.084303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084142 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01929d1a-4988-4775-b40c-915239158e7e" containerName="revoke-keys" Oct 30 14:48:21.084303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084169 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" containerName="maas-api" Oct 30 14:48:21.084303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084174 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" containerName="maas-api" Oct 30 14:48:21.084303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084218 5824 cpu_manager.go:414] "containerMap: removing container" podUID="60c9278f-54d3-49cb-975c-f49372a9fd1c" containerName="maas-api" Oct 30 14:48:21.084303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084226 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c9278f-54d3-49cb-975c-f49372a9fd1c" containerName="maas-api" Oct 30 14:48:21.084512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084322 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="01929d1a-4988-4775-b40c-915239158e7e" containerName="revoke-keys" Oct 30 14:48:21.084512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084335 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" containerName="maas-api" Oct 30 14:48:21.084512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084345 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c9278f-54d3-49cb-975c-f49372a9fd1c" containerName="maas-api" Oct 30 14:48:21.084512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.084353 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce1b7ad-c03a-457d-9f05-100e28d5886e" containerName="istio-proxy" Oct 30 14:48:21.088929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.088910 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.106138 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.106116 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf"] Oct 30 14:48:21.197254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.197220 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:48:21.197421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.197333 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:48:21.265450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.265385 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8kz\" (UniqueName: \"kubernetes.io/projected/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-kube-api-access-bv8kz\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.265450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.265421 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.265592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.265545 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.366708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.366684 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.366805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.366794 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8kz\" (UniqueName: \"kubernetes.io/projected/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-kube-api-access-bv8kz\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.366853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.366814 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.367395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.367376 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.369143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.369120 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.395540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.395515 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8kz\" (UniqueName: \"kubernetes.io/projected/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-kube-api-access-bv8kz\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.409296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.409275 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:21.715149 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:48:21.715112 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a70dcb_6929_4bbd_a6d7_f3a1a19d8cfd.slice/crio-ec252317e5f5f3c046cf68a14c549a0cb65b8145e4ec9bfeebdcef05dd53299e WatchSource:0}: Error finding container ec252317e5f5f3c046cf68a14c549a0cb65b8145e4ec9bfeebdcef05dd53299e: Status 404 returned error can't find the container with id ec252317e5f5f3c046cf68a14c549a0cb65b8145e4ec9bfeebdcef05dd53299e Oct 30 14:48:21.716723 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:21.716658 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf"] Oct 30 14:48:21.947745 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:21.947720 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a70dcb_6929_4bbd_a6d7_f3a1a19d8cfd.slice/crio-f892b7b72ca4db2d1216a36316bd6339c5e8b850ca4f160742fb14e13654f703.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:48:22.317880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:22.317800 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01929d1a-4988-4775-b40c-915239158e7e" path="/var/lib/kubelet/pods/01929d1a-4988-4775-b40c-915239158e7e/volumes" Oct 30 14:48:22.318330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:22.318308 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a" path="/var/lib/kubelet/pods/3892604c-04ee-4ce3-b7a1-f4cbf6cfb99a/volumes" Oct 30 14:48:22.439851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:22.439819 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" event={"ID":"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd","Type":"ContainerStarted","Data":"f892b7b72ca4db2d1216a36316bd6339c5e8b850ca4f160742fb14e13654f703"} Oct 30 14:48:22.440278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:22.439863 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" event={"ID":"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd","Type":"ContainerStarted","Data":"ec252317e5f5f3c046cf68a14c549a0cb65b8145e4ec9bfeebdcef05dd53299e"} Oct 30 14:48:22.485719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:22.485679 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" podStartSLOduration=1.4856656529999999 podStartE2EDuration="1.485665653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:22.485013752 +0000 UTC m=+3532.285519195" watchObservedRunningTime="2026-10-30 14:48:22.485665653 +0000 UTC m=+3532.286171095" Oct 30 14:48:23.448318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:23.448279 5824 generic.go:356] "Generic (PLEG): container finished" podID="16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd" containerID="f892b7b72ca4db2d1216a36316bd6339c5e8b850ca4f160742fb14e13654f703" exitCode=0 Oct 30 14:48:23.448715 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:23.448355 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" event={"ID":"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd","Type":"ContainerDied","Data":"f892b7b72ca4db2d1216a36316bd6339c5e8b850ca4f160742fb14e13654f703"} Oct 30 14:48:23.970010 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:23.969974 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" containerID="cri-o://515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9" gracePeriod=45 Oct 30 14:48:24.224910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.224845 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:48:24.403465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.403430 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-tmp-dir\") pod \"81b1292b-2202-4023-90a9-3409d4cf2481\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " Oct 30 14:48:24.403657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.403491 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-model-cache\") pod \"81b1292b-2202-4023-90a9-3409d4cf2481\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " Oct 30 14:48:24.403657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.403548 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b1292b-2202-4023-90a9-3409d4cf2481-tls-certs\" (UniqueName: \"kubernetes.io/secret/81b1292b-2202-4023-90a9-3409d4cf2481-tls-certs\") pod \"81b1292b-2202-4023-90a9-3409d4cf2481\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " Oct 30 14:48:24.403657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.403599 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-dshm\") pod \"81b1292b-2202-4023-90a9-3409d4cf2481\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " Oct 30 14:48:24.403829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.403668 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b1292b-2202-4023-90a9-3409d4cf2481-kube-api-access-n62fn\" (UniqueName: \"kubernetes.io/projected/81b1292b-2202-4023-90a9-3409d4cf2481-kube-api-access-n62fn\") pod \"81b1292b-2202-4023-90a9-3409d4cf2481\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " Oct 30 14:48:24.403829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.403718 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-home\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-home\") pod \"81b1292b-2202-4023-90a9-3409d4cf2481\" (UID: \"81b1292b-2202-4023-90a9-3409d4cf2481\") " Oct 30 14:48:24.403829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.403742 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-tmp-dir" pod "81b1292b-2202-4023-90a9-3409d4cf2481" (UID: "81b1292b-2202-4023-90a9-3409d4cf2481"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.403829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.403778 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-model-cache" pod "81b1292b-2202-4023-90a9-3409d4cf2481" (UID: "81b1292b-2202-4023-90a9-3409d4cf2481"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.404085 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.404058 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-home" pod "81b1292b-2202-4023-90a9-3409d4cf2481" (UID: "81b1292b-2202-4023-90a9-3409d4cf2481"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.404239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.404216 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.404239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.404238 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.404390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.404251 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.405813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.405785 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-dshm" pod "81b1292b-2202-4023-90a9-3409d4cf2481" (UID: "81b1292b-2202-4023-90a9-3409d4cf2481"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:24.405909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.405858 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b1292b-2202-4023-90a9-3409d4cf2481-tls-certs" pod "81b1292b-2202-4023-90a9-3409d4cf2481" (UID: "81b1292b-2202-4023-90a9-3409d4cf2481"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:24.405909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.405892 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b1292b-2202-4023-90a9-3409d4cf2481-kube-api-access-n62fn" pod "81b1292b-2202-4023-90a9-3409d4cf2481" (UID: "81b1292b-2202-4023-90a9-3409d4cf2481"). InnerVolumeSpecName "kube-api-access-n62fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:24.456776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.456744 5824 generic.go:356] "Generic (PLEG): container finished" podID="81b1292b-2202-4023-90a9-3409d4cf2481" containerID="515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9" exitCode=0 Oct 30 14:48:24.457141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.456786 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" Oct 30 14:48:24.457141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.456828 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" event={"ID":"81b1292b-2202-4023-90a9-3409d4cf2481","Type":"ContainerDied","Data":"515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9"} Oct 30 14:48:24.457141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.456869 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc" event={"ID":"81b1292b-2202-4023-90a9-3409d4cf2481","Type":"ContainerDied","Data":"edd8147a49363e6e8b2cbcf8372b64fe5eefb60a293949c77e32baeca5806d7f"} Oct 30 14:48:24.457141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.456896 5824 scope.go:122] "RemoveContainer" containerID="515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9" Oct 30 14:48:24.470597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.470574 5824 scope.go:122] "RemoveContainer" containerID="515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9" Oct 30 14:48:24.470910 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:24.470884 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9\": container with ID starting with 515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9 not found: ID does not exist" containerID="515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9" Oct 30 14:48:24.470990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.470915 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9"} err="failed to get container status \"515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9\": rpc error: code = NotFound desc = could not find container \"515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9\": container with ID starting with 515e28a78faf0a596e8a55960dadf988f68a020578158e4c9a730b3cee76c9b9 not found: ID does not exist" Oct 30 14:48:24.505300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.505243 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b1292b-2202-4023-90a9-3409d4cf2481-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.505300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.505265 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b1292b-2202-4023-90a9-3409d4cf2481-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.505300 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.505274 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62fn\" (UniqueName: \"kubernetes.io/projected/81b1292b-2202-4023-90a9-3409d4cf2481-kube-api-access-n62fn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.519285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.519262 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc"] Oct 30 14:48:24.534513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.534485 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-da78c4/premium-w2-da78c4-kserve-646b6c8fd9-csvsc"] Oct 30 14:48:24.587401 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.587377 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:24.706606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.706582 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-kube-api-access-bv8kz\" (UniqueName: \"kubernetes.io/projected/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-kube-api-access-bv8kz\") pod \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " Oct 30 14:48:24.706748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.706655 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-openshift-ca\") pod \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " Oct 30 14:48:24.706748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.706722 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-xks-ca\") pod \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\" (UID: \"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd\") " Oct 30 14:48:24.707060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.707030 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-openshift-ca" pod "16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd" (UID: "16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:24.707223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.707178 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.708682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.708658 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-xks-ca" pod "16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd" (UID: "16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:24.708769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.708697 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-kube-api-access-bv8kz" pod "16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd" (UID: "16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd"). InnerVolumeSpecName "kube-api-access-bv8kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:24.808242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.808153 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv8kz\" (UniqueName: \"kubernetes.io/projected/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-kube-api-access-bv8kz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:24.808242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:24.808181 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.467591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.467565 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" event={"ID":"16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd","Type":"ContainerDied","Data":"ec252317e5f5f3c046cf68a14c549a0cb65b8145e4ec9bfeebdcef05dd53299e"} Oct 30 14:48:25.467913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.467597 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf" Oct 30 14:48:25.467913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.467614 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec252317e5f5f3c046cf68a14c549a0cb65b8145e4ec9bfeebdcef05dd53299e" Oct 30 14:48:25.716613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.716553 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:48:25.733503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.733485 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:48:25.821629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821250 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-data\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.821629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821315 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52bf9caa-0df3-4620-b305-485bb89320ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf9caa-0df3-4620-b305-485bb89320ef-istio-podinfo\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.821629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821364 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-envoy\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.821629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821460 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-kube-api-access-xtllg\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-kube-api-access-xtllg\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.821629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821533 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-certs\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.821629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821578 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-istio-token\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-istio-token\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.821629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821623 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-socket\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.823332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821657 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-credential-socket\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.823332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.821784 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bf9caa-0df3-4620-b305-485bb89320ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf9caa-0df3-4620-b305-485bb89320ef-istiod-ca-cert\") pod \"52bf9caa-0df3-4620-b305-485bb89320ef\" (UID: \"52bf9caa-0df3-4620-b305-485bb89320ef\") " Oct 30 14:48:25.824083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.823806 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bf9caa-0df3-4620-b305-485bb89320ef-istiod-ca-cert" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:25.824290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.824269 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-data" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.824623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.824600 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-certs" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.825228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.825134 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-socket" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.825580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.825315 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-credential-socket" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.827716 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.827682 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-kube-api-access-xtllg" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "kube-api-access-xtllg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:25.828037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.828002 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-istio-token" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:25.830244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.829449 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-envoy" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.830244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.829880 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52bf9caa-0df3-4620-b305-485bb89320ef-istio-podinfo" pod "52bf9caa-0df3-4620-b305-485bb89320ef" (UID: "52bf9caa-0df3-4620-b305-485bb89320ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:25.924280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924249 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-podinfo\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924378 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-certs\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924478 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-socket\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924556 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-data\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924614 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f073fd1c-086a-4336-80e4-c3699dd224ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f073fd1c-086a-4336-80e4-c3699dd224ad-istiod-ca-cert\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924667 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-kube-api-access-9vq7r\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-kube-api-access-9vq7r\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924739 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-token\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-token\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924768 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-credential-socket\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924802 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-envoy\") pod \"f073fd1c-086a-4336-80e4-c3699dd224ad\" (UID: \"f073fd1c-086a-4336-80e4-c3699dd224ad\") " Oct 30 14:48:25.924823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.924800 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-certs" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.925084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925065 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f073fd1c-086a-4336-80e4-c3699dd224ad-istiod-ca-cert" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:25.925469 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925444 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-socket" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.925578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925494 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bf9caa-0df3-4620-b305-485bb89320ef-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925514 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f073fd1c-086a-4336-80e4-c3699dd224ad-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925527 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925540 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bf9caa-0df3-4620-b305-485bb89320ef-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925554 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925569 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtllg\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-kube-api-access-xtllg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925585 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925598 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925610 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bf9caa-0df3-4620-b305-485bb89320ef-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925622 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.925898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925637 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bf9caa-0df3-4620-b305-485bb89320ef-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:25.926148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.925960 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-credential-socket" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.926413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.926389 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-data" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.927986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.927958 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-envoy" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:25.928088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.928059 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-token" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:25.928391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.928366 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-podinfo" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:25.929227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:25.929182 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-kube-api-access-9vq7r" pod "f073fd1c-086a-4336-80e4-c3699dd224ad" (UID: "f073fd1c-086a-4336-80e4-c3699dd224ad"). InnerVolumeSpecName "kube-api-access-9vq7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:26.026516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.026486 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.026516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.026516 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.026698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.026532 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vq7r\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-kube-api-access-9vq7r\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.026698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.026548 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.026698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.026561 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.026698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.026573 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.026698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.026586 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f073fd1c-086a-4336-80e4-c3699dd224ad-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:26.326778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.326706 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" path="/var/lib/kubelet/pods/81b1292b-2202-4023-90a9-3409d4cf2481/volumes" Oct 30 14:48:26.475773 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.475744 5824 generic.go:356] "Generic (PLEG): container finished" podID="52bf9caa-0df3-4620-b305-485bb89320ef" containerID="9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723" exitCode=0 Oct 30 14:48:26.476174 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.475804 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" Oct 30 14:48:26.476174 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.475827 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" event={"ID":"52bf9caa-0df3-4620-b305-485bb89320ef","Type":"ContainerDied","Data":"9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723"} Oct 30 14:48:26.476174 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.475906 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f" event={"ID":"52bf9caa-0df3-4620-b305-485bb89320ef","Type":"ContainerDied","Data":"0a824398f7d077c4426837f23aa976b67782d4890194e8a0ed9e0ca0646b51a8"} Oct 30 14:48:26.476174 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.475942 5824 scope.go:122] "RemoveContainer" containerID="9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723" Oct 30 14:48:26.478096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.478073 5824 generic.go:356] "Generic (PLEG): container finished" podID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerID="0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d" exitCode=0 Oct 30 14:48:26.478377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.478169 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" Oct 30 14:48:26.478377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.478211 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" event={"ID":"f073fd1c-086a-4336-80e4-c3699dd224ad","Type":"ContainerDied","Data":"0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d"} Oct 30 14:48:26.478377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.478248 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j" event={"ID":"f073fd1c-086a-4336-80e4-c3699dd224ad","Type":"ContainerDied","Data":"b887060e73b6e0e562f83872dcb6da5faba5e53df106ab37aa4d3e3693a76cf0"} Oct 30 14:48:26.487663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.487644 5824 scope.go:122] "RemoveContainer" containerID="9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723" Oct 30 14:48:26.487927 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:26.487905 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723\": container with ID starting with 9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723 not found: ID does not exist" containerID="9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723" Oct 30 14:48:26.488003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.487938 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723"} err="failed to get container status \"9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723\": rpc error: code = NotFound desc = could not find container \"9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723\": container with ID starting with 9f1134a9953baea58272cbf29c5dcb7a3c5c8843612010ec4d578fa8c9435723 not found: ID does not exist" Oct 30 14:48:26.488003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.487964 5824 scope.go:122] "RemoveContainer" containerID="0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d" Oct 30 14:48:26.498239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.498222 5824 scope.go:122] "RemoveContainer" containerID="0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d" Oct 30 14:48:26.498479 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:26.498458 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d\": container with ID starting with 0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d not found: ID does not exist" containerID="0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d" Oct 30 14:48:26.498539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.498485 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d"} err="failed to get container status \"0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d\": rpc error: code = NotFound desc = could not find container \"0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d\": container with ID starting with 0023f3ab83e5d6812af9d5c57785a38cae99ed58f162acea8beeaebc3a29428d not found: ID does not exist" Oct 30 14:48:26.536437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.536417 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f"] Oct 30 14:48:26.549591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.549565 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-2ms7f"] Oct 30 14:48:26.581151 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.581094 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j"] Oct 30 14:48:26.586669 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:26.586650 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d05e21-openshift-default-6c99db878c-97f9j"] Oct 30 14:48:27.233227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:27.233167 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" podUID="09587b10-c0dd-49ca-be55-ba1ba847c2b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:48:27.233517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:27.233492 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" podUID="79f5cdfb-5ad6-48a6-93c3-bbdeae520169" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:48:28.317722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:28.317692 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" path="/var/lib/kubelet/pods/52bf9caa-0df3-4620-b305-485bb89320ef/volumes" Oct 30 14:48:28.318184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:28.318166 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" path="/var/lib/kubelet/pods/f073fd1c-086a-4336-80e4-c3699dd224ad/volumes" Oct 30 14:48:29.232136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232104 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p"] Oct 30 14:48:29.232796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232773 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" Oct 30 14:48:29.232796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232794 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232814 5824 cpu_manager.go:414] "containerMap: removing container" podUID="16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd" containerName="revoke-keys" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232819 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd" containerName="revoke-keys" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232829 5824 cpu_manager.go:414] "containerMap: removing container" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232834 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232856 5824 cpu_manager.go:414] "containerMap: removing container" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232860 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232953 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b1292b-2202-4023-90a9-3409d4cf2481" containerName="main" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232966 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f073fd1c-086a-4336-80e4-c3699dd224ad" containerName="istio-proxy" Oct 30 14:48:29.232971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232974 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd" containerName="revoke-keys" Oct 30 14:48:29.233499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.232982 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bf9caa-0df3-4620-b305-485bb89320ef" containerName="istio-proxy" Oct 30 14:48:29.237961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.237940 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.259217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.259182 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p"] Oct 30 14:48:29.262669 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.262645 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.262778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.262695 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.262845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.262828 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqv5\" (UniqueName: \"kubernetes.io/projected/e4dd1f27-0583-44a2-a06b-b812dbf8d637-kube-api-access-zrqv5\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.269007 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.268988 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:48:29.269152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.269134 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:48:29.363673 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.363645 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqv5\" (UniqueName: \"kubernetes.io/projected/e4dd1f27-0583-44a2-a06b-b812dbf8d637-kube-api-access-zrqv5\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.364028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.363993 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.364142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.364123 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.364697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.364674 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.366180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.366161 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.389785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.389760 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqv5\" (UniqueName: \"kubernetes.io/projected/e4dd1f27-0583-44a2-a06b-b812dbf8d637-kube-api-access-zrqv5\") pod \"maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.551535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.551474 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:29.857588 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:48:29.857553 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dd1f27_0583_44a2_a06b_b812dbf8d637.slice/crio-74e1f8b54d84685c7a80b9f546ed1b50455bee79d3b608d98aa70d08f26f0c2a WatchSource:0}: Error finding container 74e1f8b54d84685c7a80b9f546ed1b50455bee79d3b608d98aa70d08f26f0c2a: Status 404 returned error can't find the container with id 74e1f8b54d84685c7a80b9f546ed1b50455bee79d3b608d98aa70d08f26f0c2a Oct 30 14:48:29.858274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:29.858247 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p"] Oct 30 14:48:30.511597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:30.511565 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" event={"ID":"e4dd1f27-0583-44a2-a06b-b812dbf8d637","Type":"ContainerStarted","Data":"e373122c6b253ce6d47c7d45f0d9418d4f73267e3886356986a7b43c6e5c3c28"} Oct 30 14:48:30.511943 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:30.511621 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" event={"ID":"e4dd1f27-0583-44a2-a06b-b812dbf8d637","Type":"ContainerStarted","Data":"74e1f8b54d84685c7a80b9f546ed1b50455bee79d3b608d98aa70d08f26f0c2a"} Oct 30 14:48:30.561062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:30.560998 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" podStartSLOduration=1.560980464 podStartE2EDuration="1.560980464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:30.555521223 +0000 UTC m=+3540.356026667" watchObservedRunningTime="2026-10-30 14:48:30.560980464 +0000 UTC m=+3540.361485908" Oct 30 14:48:31.520164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:31.520124 5824 generic.go:356] "Generic (PLEG): container finished" podID="e4dd1f27-0583-44a2-a06b-b812dbf8d637" containerID="e373122c6b253ce6d47c7d45f0d9418d4f73267e3886356986a7b43c6e5c3c28" exitCode=0 Oct 30 14:48:31.520629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:31.520218 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" event={"ID":"e4dd1f27-0583-44a2-a06b-b812dbf8d637","Type":"ContainerDied","Data":"e373122c6b253ce6d47c7d45f0d9418d4f73267e3886356986a7b43c6e5c3c28"} Oct 30 14:48:32.452835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.452806 5824 scope.go:122] "RemoveContainer" containerID="75db56c6bb3e02c12002212e99a10f2eefed33f8adfef1d1101bb10d9ad96207" Oct 30 14:48:32.462556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.462533 5824 scope.go:122] "RemoveContainer" containerID="18e5972ef39a2376561ded34314c9a02b31e57d9518b2fc0aa1ef3310c571196" Oct 30 14:48:32.471310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.471291 5824 scope.go:122] "RemoveContainer" containerID="61bbd7bb8bbfe45d698c4f6ea398f2b036ee18bbc571413deaba7c90daa67066" Oct 30 14:48:32.683883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.683851 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:32.701962 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.701935 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4dd1f27-0583-44a2-a06b-b812dbf8d637-kube-api-access-zrqv5\" (UniqueName: \"kubernetes.io/projected/e4dd1f27-0583-44a2-a06b-b812dbf8d637-kube-api-access-zrqv5\") pod \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " Oct 30 14:48:32.702117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.702031 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-openshift-ca\") pod \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " Oct 30 14:48:32.702255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.702238 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-xks-ca\") pod \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\" (UID: \"e4dd1f27-0583-44a2-a06b-b812dbf8d637\") " Oct 30 14:48:32.702379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.702355 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-openshift-ca" pod "e4dd1f27-0583-44a2-a06b-b812dbf8d637" (UID: "e4dd1f27-0583-44a2-a06b-b812dbf8d637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:32.702708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.702689 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:32.704239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.704147 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dd1f27-0583-44a2-a06b-b812dbf8d637-kube-api-access-zrqv5" pod "e4dd1f27-0583-44a2-a06b-b812dbf8d637" (UID: "e4dd1f27-0583-44a2-a06b-b812dbf8d637"). InnerVolumeSpecName "kube-api-access-zrqv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:32.704680 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.704521 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-xks-ca" pod "e4dd1f27-0583-44a2-a06b-b812dbf8d637" (UID: "e4dd1f27-0583-44a2-a06b-b812dbf8d637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:32.803945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.803916 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4dd1f27-0583-44a2-a06b-b812dbf8d637-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:32.804088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:32.803948 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrqv5\" (UniqueName: \"kubernetes.io/projected/e4dd1f27-0583-44a2-a06b-b812dbf8d637-kube-api-access-zrqv5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:33.536933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:33.536905 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" Oct 30 14:48:33.537137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:33.536952 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p" event={"ID":"e4dd1f27-0583-44a2-a06b-b812dbf8d637","Type":"ContainerDied","Data":"74e1f8b54d84685c7a80b9f546ed1b50455bee79d3b608d98aa70d08f26f0c2a"} Oct 30 14:48:33.537137 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:33.537006 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e1f8b54d84685c7a80b9f546ed1b50455bee79d3b608d98aa70d08f26f0c2a" Oct 30 14:48:33.641995 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:33.641962 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:48:37.232301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.232257 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" podUID="09587b10-c0dd-49ca-be55-ba1ba847c2b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:48:37.233377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.233345 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" podUID="79f5cdfb-5ad6-48a6-93c3-bbdeae520169" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:48:37.944750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.944718 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9"] Oct 30 14:48:37.945693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.945658 5824 cpu_manager.go:414] "containerMap: removing container" podUID="e4dd1f27-0583-44a2-a06b-b812dbf8d637" containerName="revoke-keys" Oct 30 14:48:37.945693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.945682 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4dd1f27-0583-44a2-a06b-b812dbf8d637" containerName="revoke-keys" Oct 30 14:48:37.945869 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.945813 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4dd1f27-0583-44a2-a06b-b812dbf8d637" containerName="revoke-keys" Oct 30 14:48:37.949427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.949408 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:37.967617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.967590 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9"] Oct 30 14:48:37.977583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.977557 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:48:37.977725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:37.977566 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:48:38.064083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.064054 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2qn\" (UniqueName: \"kubernetes.io/projected/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-kube-api-access-qg2qn\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.064282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.064101 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.064368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.064287 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.165578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.165540 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2qn\" (UniqueName: \"kubernetes.io/projected/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-kube-api-access-qg2qn\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.165578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.165579 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.165799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.165652 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.166294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.166273 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.168043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.168016 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.191251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.191229 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2qn\" (UniqueName: \"kubernetes.io/projected/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-kube-api-access-qg2qn\") pod \"maas-api-revoke-sub-2f9e65e3-7lpl9\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.264493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.264439 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:38.570127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:38.570097 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9"] Oct 30 14:48:38.570837 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:48:38.570800 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b1d0d4_4f9a_4d77_9656_9dcd3167c380.slice/crio-7c020269223c3351e22ead990f78a8226cadb64c0f45176abfc6e16557746204 WatchSource:0}: Error finding container 7c020269223c3351e22ead990f78a8226cadb64c0f45176abfc6e16557746204: Status 404 returned error can't find the container with id 7c020269223c3351e22ead990f78a8226cadb64c0f45176abfc6e16557746204 Oct 30 14:48:39.585084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.585051 5824 generic.go:356] "Generic (PLEG): container finished" podID="63b1d0d4-4f9a-4d77-9656-9dcd3167c380" containerID="bc18c088da8a0c3caa3f741b063ad44728355115dc47a48089700d6321f57f79" exitCode=0 Oct 30 14:48:39.585424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.585130 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" event={"ID":"63b1d0d4-4f9a-4d77-9656-9dcd3167c380","Type":"ContainerDied","Data":"bc18c088da8a0c3caa3f741b063ad44728355115dc47a48089700d6321f57f79"} Oct 30 14:48:39.585424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.585213 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" event={"ID":"63b1d0d4-4f9a-4d77-9656-9dcd3167c380","Type":"ContainerStarted","Data":"7c020269223c3351e22ead990f78a8226cadb64c0f45176abfc6e16557746204"} Oct 30 14:48:39.798119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.798096 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:48:39.836159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.836137 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:48:39.885013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.884988 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09587b10-c0dd-49ca-be55-ba1ba847c2b9-kube-api-access-bmkvz\" (UniqueName: \"kubernetes.io/projected/09587b10-c0dd-49ca-be55-ba1ba847c2b9-kube-api-access-bmkvz\") pod \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\" (UID: \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\") " Oct 30 14:48:39.885135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.885073 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09587b10-c0dd-49ca-be55-ba1ba847c2b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09587b10-c0dd-49ca-be55-ba1ba847c2b9-plugins-config-volume\") pod \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\" (UID: \"09587b10-c0dd-49ca-be55-ba1ba847c2b9\") " Oct 30 14:48:39.885251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.885233 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-plugins-config-volume\") pod \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\" (UID: \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\") " Oct 30 14:48:39.885308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.885292 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-kube-api-access-n89q4\" (UniqueName: \"kubernetes.io/projected/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-kube-api-access-n89q4\") pod \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\" (UID: \"79f5cdfb-5ad6-48a6-93c3-bbdeae520169\") " Oct 30 14:48:39.885582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.885561 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09587b10-c0dd-49ca-be55-ba1ba847c2b9-plugins-config-volume" pod "09587b10-c0dd-49ca-be55-ba1ba847c2b9" (UID: "09587b10-c0dd-49ca-be55-ba1ba847c2b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:39.885655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.885631 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-plugins-config-volume" pod "79f5cdfb-5ad6-48a6-93c3-bbdeae520169" (UID: "79f5cdfb-5ad6-48a6-93c3-bbdeae520169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:39.885879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.885865 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:39.885921 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.885882 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09587b10-c0dd-49ca-be55-ba1ba847c2b9-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:39.887057 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.887031 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09587b10-c0dd-49ca-be55-ba1ba847c2b9-kube-api-access-bmkvz" pod "09587b10-c0dd-49ca-be55-ba1ba847c2b9" (UID: "09587b10-c0dd-49ca-be55-ba1ba847c2b9"). InnerVolumeSpecName "kube-api-access-bmkvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:39.887323 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.887298 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-kube-api-access-n89q4" pod "79f5cdfb-5ad6-48a6-93c3-bbdeae520169" (UID: "79f5cdfb-5ad6-48a6-93c3-bbdeae520169"). InnerVolumeSpecName "kube-api-access-n89q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:39.987455 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.987425 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89q4\" (UniqueName: \"kubernetes.io/projected/79f5cdfb-5ad6-48a6-93c3-bbdeae520169-kube-api-access-n89q4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:39.987455 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:39.987451 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmkvz\" (UniqueName: \"kubernetes.io/projected/09587b10-c0dd-49ca-be55-ba1ba847c2b9-kube-api-access-bmkvz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:40.592986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.592896 5824 generic.go:356] "Generic (PLEG): container finished" podID="79f5cdfb-5ad6-48a6-93c3-bbdeae520169" containerID="a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77" exitCode=0 Oct 30 14:48:40.592986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.592962 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" Oct 30 14:48:40.593509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.592978 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" event={"ID":"79f5cdfb-5ad6-48a6-93c3-bbdeae520169","Type":"ContainerDied","Data":"a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77"} Oct 30 14:48:40.593509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.593065 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt" event={"ID":"79f5cdfb-5ad6-48a6-93c3-bbdeae520169","Type":"ContainerDied","Data":"37045da4694d1b99469c9ff2473038b5cb8918b9a21babddf3406219f452663c"} Oct 30 14:48:40.593509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.593095 5824 scope.go:122] "RemoveContainer" containerID="a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77" Oct 30 14:48:40.594838 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.594813 5824 generic.go:356] "Generic (PLEG): container finished" podID="09587b10-c0dd-49ca-be55-ba1ba847c2b9" containerID="df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc" exitCode=0 Oct 30 14:48:40.594952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.594882 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" Oct 30 14:48:40.595016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.594955 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" event={"ID":"09587b10-c0dd-49ca-be55-ba1ba847c2b9","Type":"ContainerDied","Data":"df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc"} Oct 30 14:48:40.595078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.595026 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j" event={"ID":"09587b10-c0dd-49ca-be55-ba1ba847c2b9","Type":"ContainerDied","Data":"db6ca62f671ae32a828ba0c1dd8b76aa1e9f6b3dffa74091e77bdf727f4fe7c3"} Oct 30 14:48:40.604549 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.604531 5824 scope.go:122] "RemoveContainer" containerID="a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77" Oct 30 14:48:40.604837 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:40.604813 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77\": container with ID starting with a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77 not found: ID does not exist" containerID="a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77" Oct 30 14:48:40.604942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.604848 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77"} err="failed to get container status \"a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77\": rpc error: code = NotFound desc = could not find container \"a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77\": container with ID starting with a251a44eb3ff578ac432b7c332ec8446829dbaee595b3910fea77b0a94c2fd77 not found: ID does not exist" Oct 30 14:48:40.604942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.604873 5824 scope.go:122] "RemoveContainer" containerID="df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc" Oct 30 14:48:40.614120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.614092 5824 scope.go:122] "RemoveContainer" containerID="df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc" Oct 30 14:48:40.614481 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:40.614460 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc\": container with ID starting with df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc not found: ID does not exist" containerID="df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc" Oct 30 14:48:40.614550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.614488 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc"} err="failed to get container status \"df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc\": rpc error: code = NotFound desc = could not find container \"df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc\": container with ID starting with df9503fbb723c4da551ceb2420d3b9e06eba3f11d26fe82265dba8e20d5d4ecc not found: ID does not exist" Oct 30 14:48:40.692214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.690133 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt"] Oct 30 14:48:40.701211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.694478 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-da78c4-5bf784496d-rfcbt"] Oct 30 14:48:40.744865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.744774 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j"] Oct 30 14:48:40.748783 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.748757 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-da78c4-8c5d878ff-8hb8j"] Oct 30 14:48:40.834184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.834164 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:40.898484 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.898417 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-xks-ca\") pod \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " Oct 30 14:48:40.898629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.898518 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-kube-api-access-qg2qn\" (UniqueName: \"kubernetes.io/projected/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-kube-api-access-qg2qn\") pod \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " Oct 30 14:48:40.898629 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.898558 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-openshift-ca\") pod \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\" (UID: \"63b1d0d4-4f9a-4d77-9656-9dcd3167c380\") " Oct 30 14:48:40.899016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.898985 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-openshift-ca" pod "63b1d0d4-4f9a-4d77-9656-9dcd3167c380" (UID: "63b1d0d4-4f9a-4d77-9656-9dcd3167c380"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:40.900544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.900517 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-xks-ca" pod "63b1d0d4-4f9a-4d77-9656-9dcd3167c380" (UID: "63b1d0d4-4f9a-4d77-9656-9dcd3167c380"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:40.900651 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:40.900577 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-kube-api-access-qg2qn" pod "63b1d0d4-4f9a-4d77-9656-9dcd3167c380" (UID: "63b1d0d4-4f9a-4d77-9656-9dcd3167c380"). InnerVolumeSpecName "kube-api-access-qg2qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:41.000176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:41.000137 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg2qn\" (UniqueName: \"kubernetes.io/projected/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-kube-api-access-qg2qn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:41.000176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:41.000175 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:41.000405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:41.000227 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b1d0d4-4f9a-4d77-9656-9dcd3167c380-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:41.608415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:41.608384 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" event={"ID":"63b1d0d4-4f9a-4d77-9656-9dcd3167c380","Type":"ContainerDied","Data":"7c020269223c3351e22ead990f78a8226cadb64c0f45176abfc6e16557746204"} Oct 30 14:48:41.608803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:41.608432 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c020269223c3351e22ead990f78a8226cadb64c0f45176abfc6e16557746204" Oct 30 14:48:41.608803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:41.608436 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9" Oct 30 14:48:42.318049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:42.318016 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09587b10-c0dd-49ca-be55-ba1ba847c2b9" path="/var/lib/kubelet/pods/09587b10-c0dd-49ca-be55-ba1ba847c2b9/volumes" Oct 30 14:48:42.318456 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:42.318438 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f5cdfb-5ad6-48a6-93c3-bbdeae520169" path="/var/lib/kubelet/pods/79f5cdfb-5ad6-48a6-93c3-bbdeae520169/volumes" Oct 30 14:48:45.140535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:45.140506 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp"] Oct 30 14:48:45.140956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:45.140731 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" containerID="cri-o://044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e" gracePeriod=30 Oct 30 14:48:50.412787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.412724 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:48:50.532643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.532613 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fb1faa-2773-4fde-982d-98e73d5dec84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb1faa-2773-4fde-982d-98e73d5dec84-istiod-ca-cert\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.532832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.532662 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-data\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.532832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.532693 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-credential-socket\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.532832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.532764 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-certs\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.532832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.532808 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-envoy\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.533139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.532841 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-token\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-token\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.533139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.532966 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-kube-api-access-4nd8p\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.533139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533040 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-data" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:50.533139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533054 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-credential-socket" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:50.533139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533106 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fb1faa-2773-4fde-982d-98e73d5dec84-istiod-ca-cert" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:50.533139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533112 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-socket\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.533539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533300 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-podinfo\") pod \"69fb1faa-2773-4fde-982d-98e73d5dec84\" (UID: \"69fb1faa-2773-4fde-982d-98e73d5dec84\") " Oct 30 14:48:50.533539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533327 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-socket" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:50.533539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533361 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-certs" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:50.533835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533816 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.533930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533837 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fb1faa-2773-4fde-982d-98e73d5dec84-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.533930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533847 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.533930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533858 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.533930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.533871 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.535240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.535209 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-podinfo" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:48:50.535347 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.535308 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-token" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:50.535530 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.535511 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-kube-api-access-4nd8p" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "kube-api-access-4nd8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:50.535572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.535546 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-envoy" pod "69fb1faa-2773-4fde-982d-98e73d5dec84" (UID: "69fb1faa-2773-4fde-982d-98e73d5dec84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:50.637222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.637145 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nd8p\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-kube-api-access-4nd8p\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.637222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.637203 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.637222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.637222 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.637555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.637235 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fb1faa-2773-4fde-982d-98e73d5dec84-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:50.678741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.678704 5824 generic.go:356] "Generic (PLEG): container finished" podID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerID="044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e" exitCode=0 Oct 30 14:48:50.678906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.678745 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" Oct 30 14:48:50.678906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.678791 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" event={"ID":"69fb1faa-2773-4fde-982d-98e73d5dec84","Type":"ContainerDied","Data":"044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e"} Oct 30 14:48:50.678906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.678849 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp" event={"ID":"69fb1faa-2773-4fde-982d-98e73d5dec84","Type":"ContainerDied","Data":"21bc713bfdfa57569c910801cd4d5af62278e479ad0a78fb4c32480e536c235b"} Oct 30 14:48:50.678906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.678877 5824 scope.go:122] "RemoveContainer" containerID="044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e" Oct 30 14:48:50.690413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.690390 5824 scope.go:122] "RemoveContainer" containerID="044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e" Oct 30 14:48:50.690683 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:50.690665 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e\": container with ID starting with 044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e not found: ID does not exist" containerID="044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e" Oct 30 14:48:50.690759 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.690691 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e"} err="failed to get container status \"044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e\": rpc error: code = NotFound desc = could not find container \"044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e\": container with ID starting with 044b8feb51693b9772321ff2b9c6ebfa43851ad4dbbe2d7cd887c3c96ba6a94e not found: ID does not exist" Oct 30 14:48:50.756638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.756611 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp"] Oct 30 14:48:50.763215 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:50.763140 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a59ecb-openshift-default-6ffc4474c9-kh8cp"] Oct 30 14:48:52.317271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:52.317230 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" path="/var/lib/kubelet/pods/69fb1faa-2773-4fde-982d-98e73d5dec84/volumes" Oct 30 14:48:55.244889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:55.244840 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf"] Oct 30 14:48:55.261909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:55.261873 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9e3aa0-mrgpf"] Oct 30 14:48:56.317632 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:56.317601 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd" path="/var/lib/kubelet/pods/16a70dcb-6929-4bbd-a6d7-f3a1a19d8cfd/volumes" Oct 30 14:48:57.657726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:57.657690 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p"] Oct 30 14:48:57.676918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:57.676877 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bcc710-z9l2p"] Oct 30 14:48:58.317799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:58.317767 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4dd1f27-0583-44a2-a06b-b812dbf8d637" path="/var/lib/kubelet/pods/e4dd1f27-0583-44a2-a06b-b812dbf8d637/volumes" Oct 30 14:48:59.281986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.281950 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z"] Oct 30 14:48:59.282509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.282452 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" podUID="c3eebbff-a656-4bc8-a369-3da7d76830f7" containerName="maas-api" containerID="cri-o://e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc" gracePeriod=30 Oct 30 14:48:59.575863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.575839 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:48:59.696113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.696081 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls\") pod \"c3eebbff-a656-4bc8-a369-3da7d76830f7\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " Oct 30 14:48:59.696306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.696125 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3eebbff-a656-4bc8-a369-3da7d76830f7-kube-api-access-nbkt7\" (UniqueName: \"kubernetes.io/projected/c3eebbff-a656-4bc8-a369-3da7d76830f7-kube-api-access-nbkt7\") pod \"c3eebbff-a656-4bc8-a369-3da7d76830f7\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " Oct 30 14:48:59.696306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.696151 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3eebbff-a656-4bc8-a369-3da7d76830f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3eebbff-a656-4bc8-a369-3da7d76830f7-openshift-service-ca\") pod \"c3eebbff-a656-4bc8-a369-3da7d76830f7\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " Oct 30 14:48:59.696306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.696185 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-metrics-certs\") pod \"c3eebbff-a656-4bc8-a369-3da7d76830f7\" (UID: \"c3eebbff-a656-4bc8-a369-3da7d76830f7\") " Oct 30 14:48:59.696524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.696499 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3eebbff-a656-4bc8-a369-3da7d76830f7-openshift-service-ca" pod "c3eebbff-a656-4bc8-a369-3da7d76830f7" (UID: "c3eebbff-a656-4bc8-a369-3da7d76830f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:59.696705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.696671 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3eebbff-a656-4bc8-a369-3da7d76830f7-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:59.698176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.698148 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls" pod "c3eebbff-a656-4bc8-a369-3da7d76830f7" (UID: "c3eebbff-a656-4bc8-a369-3da7d76830f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:59.698176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.698164 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3eebbff-a656-4bc8-a369-3da7d76830f7-kube-api-access-nbkt7" pod "c3eebbff-a656-4bc8-a369-3da7d76830f7" (UID: "c3eebbff-a656-4bc8-a369-3da7d76830f7"). InnerVolumeSpecName "kube-api-access-nbkt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:59.698337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.698170 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-metrics-certs" pod "c3eebbff-a656-4bc8-a369-3da7d76830f7" (UID: "c3eebbff-a656-4bc8-a369-3da7d76830f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:59.742956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.742928 5824 generic.go:356] "Generic (PLEG): container finished" podID="c3eebbff-a656-4bc8-a369-3da7d76830f7" containerID="e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc" exitCode=0 Oct 30 14:48:59.743068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.742977 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" Oct 30 14:48:59.743068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.743008 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" event={"ID":"c3eebbff-a656-4bc8-a369-3da7d76830f7","Type":"ContainerDied","Data":"e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc"} Oct 30 14:48:59.743068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.743059 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z" event={"ID":"c3eebbff-a656-4bc8-a369-3da7d76830f7","Type":"ContainerDied","Data":"c479d3f2d6311e95c4da8037abcfcde306c6e785f95c50716d30ab54a0bd05a9"} Oct 30 14:48:59.743229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.743081 5824 scope.go:122] "RemoveContainer" containerID="e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc" Oct 30 14:48:59.753793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.753775 5824 scope.go:122] "RemoveContainer" containerID="e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc" Oct 30 14:48:59.754101 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:48:59.754084 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc\": container with ID starting with e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc not found: ID does not exist" containerID="e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc" Oct 30 14:48:59.754161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.754110 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc"} err="failed to get container status \"e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc\": rpc error: code = NotFound desc = could not find container \"e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc\": container with ID starting with e9b0203f12d49f94cb62eb692d1f16d3e3eeaaf19cf93cc32d273c682a7414bc not found: ID does not exist" Oct 30 14:48:59.810274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.806828 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:59.810274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.806862 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbkt7\" (UniqueName: \"kubernetes.io/projected/c3eebbff-a656-4bc8-a369-3da7d76830f7-kube-api-access-nbkt7\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:59.810274 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.806897 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3eebbff-a656-4bc8-a369-3da7d76830f7-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:48:59.845989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.845952 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z"] Oct 30 14:48:59.850303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:48:59.850272 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3b95dd-c58b97489-4s94z"] Oct 30 14:49:00.318746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:00.318717 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3eebbff-a656-4bc8-a369-3da7d76830f7" path="/var/lib/kubelet/pods/c3eebbff-a656-4bc8-a369-3da7d76830f7/volumes" Oct 30 14:49:01.475204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:01.475150 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v"] Oct 30 14:49:01.509924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:01.509888 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh"] Oct 30 14:49:03.642254 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:03.642220 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:49:06.385950 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.385916 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb"] Oct 30 14:49:06.386516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386499 5824 cpu_manager.go:414] "containerMap: removing container" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" Oct 30 14:49:06.386558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386516 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" Oct 30 14:49:06.386558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386530 5824 cpu_manager.go:414] "containerMap: removing container" podUID="09587b10-c0dd-49ca-be55-ba1ba847c2b9" containerName="payload-pre-processing" Oct 30 14:49:06.386558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386535 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09587b10-c0dd-49ca-be55-ba1ba847c2b9" containerName="payload-pre-processing" Oct 30 14:49:06.386558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386545 5824 cpu_manager.go:414] "containerMap: removing container" podUID="63b1d0d4-4f9a-4d77-9656-9dcd3167c380" containerName="revoke-keys" Oct 30 14:49:06.386558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386550 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b1d0d4-4f9a-4d77-9656-9dcd3167c380" containerName="revoke-keys" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386561 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c3eebbff-a656-4bc8-a369-3da7d76830f7" containerName="maas-api" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386566 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3eebbff-a656-4bc8-a369-3da7d76830f7" containerName="maas-api" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386594 5824 cpu_manager.go:414] "containerMap: removing container" podUID="79f5cdfb-5ad6-48a6-93c3-bbdeae520169" containerName="payload-processing" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386599 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f5cdfb-5ad6-48a6-93c3-bbdeae520169" containerName="payload-processing" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386688 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f5cdfb-5ad6-48a6-93c3-bbdeae520169" containerName="payload-processing" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386702 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fb1faa-2773-4fde-982d-98e73d5dec84" containerName="istio-proxy" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386708 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="09587b10-c0dd-49ca-be55-ba1ba847c2b9" containerName="payload-pre-processing" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386714 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b1d0d4-4f9a-4d77-9656-9dcd3167c380" containerName="revoke-keys" Oct 30 14:49:06.386733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.386719 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3eebbff-a656-4bc8-a369-3da7d76830f7" containerName="maas-api" Oct 30 14:49:06.390158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.390138 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.408437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.408413 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb"] Oct 30 14:49:06.418044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.418023 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:49:06.418167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.418109 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:49:06.475452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.475419 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" podUID="c1711c3e-b183-4985-8d8a-7a4da293cf1d" containerName="payload-processing" containerID="cri-o://134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae" gracePeriod=25 Oct 30 14:49:06.488158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.488131 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.488334 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.488319 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.488395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.488380 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6nv\" (UniqueName: \"kubernetes.io/projected/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-kube-api-access-sm6nv\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.510823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.510797 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" podUID="63b417b1-8f7a-421b-8db2-4fa9ac539d94" containerName="payload-pre-processing" containerID="cri-o://a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78" gracePeriod=25 Oct 30 14:49:06.589684 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.589651 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.589851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.589740 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6nv\" (UniqueName: \"kubernetes.io/projected/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-kube-api-access-sm6nv\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.589851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.589829 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.590661 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.590635 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.593988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.593964 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.616172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.616146 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6nv\" (UniqueName: \"kubernetes.io/projected/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-kube-api-access-sm6nv\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:06.703852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:06.703825 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:07.009752 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:49:07.009724 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2e078c_bc3b_41ea_8b90_7bf3b6c3e653.slice/crio-b6474e5240809e86c825caf75a04ceb43b90f410cd3e8deba6437d4f00a4690a WatchSource:0}: Error finding container b6474e5240809e86c825caf75a04ceb43b90f410cd3e8deba6437d4f00a4690a: Status 404 returned error can't find the container with id b6474e5240809e86c825caf75a04ceb43b90f410cd3e8deba6437d4f00a4690a Oct 30 14:49:07.010237 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:07.010119 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb"] Oct 30 14:49:07.810609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:07.810574 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" event={"ID":"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653","Type":"ContainerStarted","Data":"2a48b6b33f5f44857021bafb507609f5bcd7ace18b03632a2393f1156a5f9c26"} Oct 30 14:49:07.810969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:07.810620 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" event={"ID":"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653","Type":"ContainerStarted","Data":"b6474e5240809e86c825caf75a04ceb43b90f410cd3e8deba6437d4f00a4690a"} Oct 30 14:49:07.855443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:07.855400 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" podStartSLOduration=1.855386057 podStartE2EDuration="1.855386057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:49:07.85413056 +0000 UTC m=+3577.654636013" watchObservedRunningTime="2026-10-30 14:49:07.855386057 +0000 UTC m=+3577.655891496" Oct 30 14:49:08.818521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:08.818484 5824 generic.go:356] "Generic (PLEG): container finished" podID="0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653" containerID="2a48b6b33f5f44857021bafb507609f5bcd7ace18b03632a2393f1156a5f9c26" exitCode=0 Oct 30 14:49:08.818899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:08.818613 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" event={"ID":"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653","Type":"ContainerDied","Data":"2a48b6b33f5f44857021bafb507609f5bcd7ace18b03632a2393f1156a5f9c26"} Oct 30 14:49:09.987745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:09.987720 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:10.044023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.043991 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-kube-api-access-sm6nv\" (UniqueName: \"kubernetes.io/projected/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-kube-api-access-sm6nv\") pod \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " Oct 30 14:49:10.044176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.044088 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-xks-ca\") pod \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " Oct 30 14:49:10.044275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.044225 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-openshift-ca\") pod \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\" (UID: \"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653\") " Oct 30 14:49:10.044798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.044768 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-openshift-ca" pod "0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653" (UID: "0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:10.046520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.046482 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-kube-api-access-sm6nv" pod "0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653" (UID: "0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653"). InnerVolumeSpecName "kube-api-access-sm6nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:10.048091 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.048065 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-xks-ca" pod "0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653" (UID: "0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:10.145898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.145835 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6nv\" (UniqueName: \"kubernetes.io/projected/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-kube-api-access-sm6nv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:10.145898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.145859 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:10.145898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.145871 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:10.834980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.834950 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" Oct 30 14:49:10.835168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.834990 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb" event={"ID":"0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653","Type":"ContainerDied","Data":"b6474e5240809e86c825caf75a04ceb43b90f410cd3e8deba6437d4f00a4690a"} Oct 30 14:49:10.835168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:10.835072 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6474e5240809e86c825caf75a04ceb43b90f410cd3e8deba6437d4f00a4690a" Oct 30 14:49:11.203894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:11.203853 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" podUID="c1711c3e-b183-4985-8d8a-7a4da293cf1d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:11.204297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:11.203911 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" podUID="63b417b1-8f7a-421b-8db2-4fa9ac539d94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:14.071412 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.071381 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv"] Oct 30 14:49:14.071801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.071578 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" containerID="cri-o://1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf" gracePeriod=30 Oct 30 14:49:14.176630 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.176595 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs"] Oct 30 14:49:14.176818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.176788 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" containerID="cri-o://48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595" gracePeriod=30 Oct 30 14:49:14.713143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.713112 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4"] Oct 30 14:49:14.713818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.713800 5824 cpu_manager.go:414] "containerMap: removing container" podUID="0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653" containerName="revoke-keys" Oct 30 14:49:14.713818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.713818 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653" containerName="revoke-keys" Oct 30 14:49:14.713937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.713918 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653" containerName="revoke-keys" Oct 30 14:49:14.717495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.717474 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.735463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.735438 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4"] Oct 30 14:49:14.735954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.735931 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:49:14.741495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.741470 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:49:14.801560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.801528 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg8g2\" (UniqueName: \"kubernetes.io/projected/abc14879-32af-4e94-bd53-7475533913d8-kube-api-access-wg8g2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.801696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.801562 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc14879-32af-4e94-bd53-7475533913d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.801762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.801699 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc14879-32af-4e94-bd53-7475533913d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.902678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.902649 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg8g2\" (UniqueName: \"kubernetes.io/projected/abc14879-32af-4e94-bd53-7475533913d8-kube-api-access-wg8g2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.902794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.902683 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc14879-32af-4e94-bd53-7475533913d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.902794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.902738 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc14879-32af-4e94-bd53-7475533913d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.903479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.903461 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc14879-32af-4e94-bd53-7475533913d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.905083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.905065 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc14879-32af-4e94-bd53-7475533913d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:14.929278 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:14.929258 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg8g2\" (UniqueName: \"kubernetes.io/projected/abc14879-32af-4e94-bd53-7475533913d8-kube-api-access-wg8g2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:15.030620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:15.030558 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:15.345367 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:49:15.345338 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc14879_32af_4e94_bd53_7475533913d8.slice/crio-8bc8c8611f1903cdebcd346ddb4ee9fba3c900d275f8410c06c2e798ddee91f0 WatchSource:0}: Error finding container 8bc8c8611f1903cdebcd346ddb4ee9fba3c900d275f8410c06c2e798ddee91f0: Status 404 returned error can't find the container with id 8bc8c8611f1903cdebcd346ddb4ee9fba3c900d275f8410c06c2e798ddee91f0 Oct 30 14:49:15.346518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:15.346488 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4"] Oct 30 14:49:15.873813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:15.873768 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" event={"ID":"abc14879-32af-4e94-bd53-7475533913d8","Type":"ContainerStarted","Data":"241e3e99acdddc893166b602dcfb33441e7a113aa16cc397d240e0915efd6794"} Oct 30 14:49:15.874026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:15.873834 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" event={"ID":"abc14879-32af-4e94-bd53-7475533913d8","Type":"ContainerStarted","Data":"8bc8c8611f1903cdebcd346ddb4ee9fba3c900d275f8410c06c2e798ddee91f0"} Oct 30 14:49:15.922624 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:15.922576 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" podStartSLOduration=1.922562828 podStartE2EDuration="1.922562828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:49:15.921332574 +0000 UTC m=+3585.721838018" watchObservedRunningTime="2026-10-30 14:49:15.922562828 +0000 UTC m=+3585.723068265" Oct 30 14:49:16.887732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:16.887691 5824 generic.go:356] "Generic (PLEG): container finished" podID="abc14879-32af-4e94-bd53-7475533913d8" containerID="241e3e99acdddc893166b602dcfb33441e7a113aa16cc397d240e0915efd6794" exitCode=0 Oct 30 14:49:16.888312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:16.887845 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" event={"ID":"abc14879-32af-4e94-bd53-7475533913d8","Type":"ContainerDied","Data":"241e3e99acdddc893166b602dcfb33441e7a113aa16cc397d240e0915efd6794"} Oct 30 14:49:18.058374 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.058347 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:18.141117 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.141090 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc14879-32af-4e94-bd53-7475533913d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc14879-32af-4e94-bd53-7475533913d8-maas-api-xks-ca\") pod \"abc14879-32af-4e94-bd53-7475533913d8\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " Oct 30 14:49:18.141337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.141131 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc14879-32af-4e94-bd53-7475533913d8-kube-api-access-wg8g2\" (UniqueName: \"kubernetes.io/projected/abc14879-32af-4e94-bd53-7475533913d8-kube-api-access-wg8g2\") pod \"abc14879-32af-4e94-bd53-7475533913d8\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " Oct 30 14:49:18.141337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.141253 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc14879-32af-4e94-bd53-7475533913d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc14879-32af-4e94-bd53-7475533913d8-maas-api-openshift-ca\") pod \"abc14879-32af-4e94-bd53-7475533913d8\" (UID: \"abc14879-32af-4e94-bd53-7475533913d8\") " Oct 30 14:49:18.141735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.141693 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc14879-32af-4e94-bd53-7475533913d8-maas-api-openshift-ca" pod "abc14879-32af-4e94-bd53-7475533913d8" (UID: "abc14879-32af-4e94-bd53-7475533913d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:18.141913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.141890 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abc14879-32af-4e94-bd53-7475533913d8-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:18.143362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.143337 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc14879-32af-4e94-bd53-7475533913d8-kube-api-access-wg8g2" pod "abc14879-32af-4e94-bd53-7475533913d8" (UID: "abc14879-32af-4e94-bd53-7475533913d8"). InnerVolumeSpecName "kube-api-access-wg8g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:18.143583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.143569 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc14879-32af-4e94-bd53-7475533913d8-maas-api-xks-ca" pod "abc14879-32af-4e94-bd53-7475533913d8" (UID: "abc14879-32af-4e94-bd53-7475533913d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:18.242731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.242703 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abc14879-32af-4e94-bd53-7475533913d8-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:18.242731 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.242730 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg8g2\" (UniqueName: \"kubernetes.io/projected/abc14879-32af-4e94-bd53-7475533913d8-kube-api-access-wg8g2\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:18.267106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.267075 5824 patch_prober.go:29] interesting pod/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:49:18.267265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.267138 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:49:18.904518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.904490 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" Oct 30 14:49:18.904518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.904496 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4" event={"ID":"abc14879-32af-4e94-bd53-7475533913d8","Type":"ContainerDied","Data":"8bc8c8611f1903cdebcd346ddb4ee9fba3c900d275f8410c06c2e798ddee91f0"} Oct 30 14:49:18.904737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:18.904548 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bc8c8611f1903cdebcd346ddb4ee9fba3c900d275f8410c06c2e798ddee91f0" Oct 30 14:49:19.344451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.344420 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:49:19.441952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.441898 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:49:19.455460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455439 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-socket\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.455599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455476 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-data\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.455599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455520 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-kube-api-access-5cm67\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-kube-api-access-5cm67\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.455599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455574 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-credential-socket\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.455756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455664 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-podinfo\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.455756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455702 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-envoy\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.455756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455709 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-socket" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.455911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455791 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-data" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.455911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455867 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-token\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-token\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.455911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455885 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-credential-socket" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.455911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.455899 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c61044-6bb0-41a0-9655-588f8dff0470-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c61044-6bb0-41a0-9655-588f8dff0470-istiod-ca-cert\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.456130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.456061 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-certs\") pod \"f4c61044-6bb0-41a0-9655-588f8dff0470\" (UID: \"f4c61044-6bb0-41a0-9655-588f8dff0470\") " Oct 30 14:49:19.456271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.456247 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c61044-6bb0-41a0-9655-588f8dff0470-istiod-ca-cert" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:19.456350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.456326 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-certs" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.456665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.456642 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.456779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.456669 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.456779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.456680 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.456779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.456689 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c61044-6bb0-41a0-9655-588f8dff0470-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.456779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.456698 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.457769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.457746 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-kube-api-access-5cm67" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "kube-api-access-5cm67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:19.457861 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.457846 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-token" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:19.458057 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.458041 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-podinfo" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:49:19.458143 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.458131 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-envoy" pod "f4c61044-6bb0-41a0-9655-588f8dff0470" (UID: "f4c61044-6bb0-41a0-9655-588f8dff0470"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.465515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.465493 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc"] Oct 30 14:49:19.525037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.525007 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp"] Oct 30 14:49:19.557335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557300 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-data\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.557521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557367 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-socket\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.557521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557451 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-envoy\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.557521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557517 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-token\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-token\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.557706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557557 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-credential-socket\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.557706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557664 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-certs\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.557817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557696 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-data" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.557817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557706 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-kube-api-access-vtbcm\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-kube-api-access-vtbcm\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.557817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557701 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-socket" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.558000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557844 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9054b512-4a30-4209-baf0-3f0ccd66a73d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9054b512-4a30-4209-baf0-3f0ccd66a73d-istiod-ca-cert\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.558000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.557882 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-podinfo\") pod \"9054b512-4a30-4209-baf0-3f0ccd66a73d\" (UID: \"9054b512-4a30-4209-baf0-3f0ccd66a73d\") " Oct 30 14:49:19.558330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558305 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-credential-socket" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.558642 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558619 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.558743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558647 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.558743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558662 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.558743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558677 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cm67\" (UniqueName: \"kubernetes.io/projected/f4c61044-6bb0-41a0-9655-588f8dff0470-kube-api-access-5cm67\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.558743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558693 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.558743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558706 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.558743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558720 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c61044-6bb0-41a0-9655-588f8dff0470-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.559123 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.558967 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-certs" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.559340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.559313 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9054b512-4a30-4209-baf0-3f0ccd66a73d-istiod-ca-cert" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:19.560388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.560358 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-kube-api-access-vtbcm" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "kube-api-access-vtbcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:19.560875 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.560845 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-podinfo" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:49:19.561460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.561434 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-token" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:19.561572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.561493 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-envoy" pod "9054b512-4a30-4209-baf0-3f0ccd66a73d" (UID: "9054b512-4a30-4209-baf0-3f0ccd66a73d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:19.659698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.659664 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.659698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.659692 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.659698 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.659700 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9054b512-4a30-4209-baf0-3f0ccd66a73d-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.659937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.659709 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtbcm\" (UniqueName: \"kubernetes.io/projected/9054b512-4a30-4209-baf0-3f0ccd66a73d-kube-api-access-vtbcm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.659937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.659718 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9054b512-4a30-4209-baf0-3f0ccd66a73d-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.659937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.659726 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9054b512-4a30-4209-baf0-3f0ccd66a73d-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:19.913069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.912993 5824 generic.go:356] "Generic (PLEG): container finished" podID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerID="48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595" exitCode=0 Oct 30 14:49:19.913238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.913062 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" Oct 30 14:49:19.913238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.913079 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" event={"ID":"9054b512-4a30-4209-baf0-3f0ccd66a73d","Type":"ContainerDied","Data":"48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595"} Oct 30 14:49:19.913238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.913165 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs" event={"ID":"9054b512-4a30-4209-baf0-3f0ccd66a73d","Type":"ContainerDied","Data":"fbbe5503504b82b79ce9f234dc12ba9e9e8d5981022ac80fb7903c8e7122b7cb"} Oct 30 14:49:19.913238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.913226 5824 scope.go:122] "RemoveContainer" containerID="48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595" Oct 30 14:49:19.914597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.914566 5824 generic.go:356] "Generic (PLEG): container finished" podID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerID="1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf" exitCode=0 Oct 30 14:49:19.914728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.914597 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" event={"ID":"f4c61044-6bb0-41a0-9655-588f8dff0470","Type":"ContainerDied","Data":"1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf"} Oct 30 14:49:19.914728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.914627 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" Oct 30 14:49:19.914728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.914637 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv" event={"ID":"f4c61044-6bb0-41a0-9655-588f8dff0470","Type":"ContainerDied","Data":"78460645670f8a075c96d496abd32d61aa6d0e0c79e417ecbe98f7509dd8e396"} Oct 30 14:49:19.926524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.926499 5824 scope.go:122] "RemoveContainer" containerID="48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595" Oct 30 14:49:19.926833 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:19.926803 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595\": container with ID starting with 48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595 not found: ID does not exist" containerID="48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595" Oct 30 14:49:19.926909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.926842 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595"} err="failed to get container status \"48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595\": rpc error: code = NotFound desc = could not find container \"48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595\": container with ID starting with 48629a89bd9897dc612577374a3cce00302d8b892d7c860e15d151b41d585595 not found: ID does not exist" Oct 30 14:49:19.926909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.926862 5824 scope.go:122] "RemoveContainer" containerID="1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf" Oct 30 14:49:19.936268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.936251 5824 scope.go:122] "RemoveContainer" containerID="1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf" Oct 30 14:49:19.936492 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:19.936475 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf\": container with ID starting with 1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf not found: ID does not exist" containerID="1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf" Oct 30 14:49:19.936565 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.936497 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf"} err="failed to get container status \"1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf\": rpc error: code = NotFound desc = could not find container \"1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf\": container with ID starting with 1e7e73192aa753b778d0f3a5d7fef9196d15af08c7dce36a6dba9885b5daddcf not found: ID does not exist" Oct 30 14:49:19.979240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.979218 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv"] Oct 30 14:49:19.990014 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:19.989991 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-da78c4-openshift-default-84dbb4c47f-pfpzv"] Oct 30 14:49:20.028568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:20.028544 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs"] Oct 30 14:49:20.035501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:20.035482 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9e3aa0-openshift-default-bc5d75d8f-jp6fs"] Oct 30 14:49:20.317065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:20.317029 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" path="/var/lib/kubelet/pods/9054b512-4a30-4209-baf0-3f0ccd66a73d/volumes" Oct 30 14:49:20.317511 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:20.317494 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" path="/var/lib/kubelet/pods/f4c61044-6bb0-41a0-9655-588f8dff0470/volumes" Oct 30 14:49:21.203827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:21.203792 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" podUID="63b417b1-8f7a-421b-8db2-4fa9ac539d94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:21.203827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:21.203816 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" podUID="c1711c3e-b183-4985-8d8a-7a4da293cf1d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:22.805356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.805321 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc"] Oct 30 14:49:22.805908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.805893 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" Oct 30 14:49:22.805953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.805908 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" Oct 30 14:49:22.805953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.805931 5824 cpu_manager.go:414] "containerMap: removing container" podUID="abc14879-32af-4e94-bd53-7475533913d8" containerName="revoke-keys" Oct 30 14:49:22.805953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.805936 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc14879-32af-4e94-bd53-7475533913d8" containerName="revoke-keys" Oct 30 14:49:22.806054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.805955 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" Oct 30 14:49:22.806054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.805960 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" Oct 30 14:49:22.806054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.806049 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc14879-32af-4e94-bd53-7475533913d8" containerName="revoke-keys" Oct 30 14:49:22.806146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.806062 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9054b512-4a30-4209-baf0-3f0ccd66a73d" containerName="istio-proxy" Oct 30 14:49:22.806146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.806074 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c61044-6bb0-41a0-9655-588f8dff0470" containerName="istio-proxy" Oct 30 14:49:22.811096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.811078 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:22.829678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.829648 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc"] Oct 30 14:49:22.894621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.894593 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:22.894744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.894719 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9scw5\" (UniqueName: \"kubernetes.io/projected/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-kube-api-access-9scw5\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:22.894789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.894754 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:22.920585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.920554 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:49:22.920585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.920570 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:49:22.995821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.995797 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:22.995948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.995864 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9scw5\" (UniqueName: \"kubernetes.io/projected/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-kube-api-access-9scw5\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:22.995948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.995889 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:22.996585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.996558 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:22.998111 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:22.998090 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:23.021690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:23.021671 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9scw5\" (UniqueName: \"kubernetes.io/projected/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-kube-api-access-9scw5\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:23.125331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:23.125268 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:23.432179 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:49:23.432147 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4829b5_2fc2_4337_9d3a_8fa8dc8a6e3f.slice/crio-252e4535c445db0bc5943ced0bc1918bff552e13ed5c07c2e68fb4952eeeea44 WatchSource:0}: Error finding container 252e4535c445db0bc5943ced0bc1918bff552e13ed5c07c2e68fb4952eeeea44: Status 404 returned error can't find the container with id 252e4535c445db0bc5943ced0bc1918bff552e13ed5c07c2e68fb4952eeeea44 Oct 30 14:49:23.433367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:23.433340 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc"] Oct 30 14:49:23.948890 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:23.948846 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" event={"ID":"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f","Type":"ContainerStarted","Data":"73b758e562477aa3c447bb37deb3c66a8af01eb7c4a2dd45a082b8e2d1ae9107"} Oct 30 14:49:23.949238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:23.948907 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" event={"ID":"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f","Type":"ContainerStarted","Data":"252e4535c445db0bc5943ced0bc1918bff552e13ed5c07c2e68fb4952eeeea44"} Oct 30 14:49:23.995182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:23.995134 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" podStartSLOduration=1.9951037820000002 podStartE2EDuration="1.995103782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:49:23.992268034 +0000 UTC m=+3593.792773480" watchObservedRunningTime="2026-10-30 14:49:23.995103782 +0000 UTC m=+3593.795609224" Oct 30 14:49:24.466722 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:24.466667 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" podUID="eaa3f864-0b20-41ca-9594-61803ab6411f" containerName="payload-processing" containerID="cri-o://5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a" gracePeriod=25 Oct 30 14:49:24.525328 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:24.525251 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" podUID="f399c067-eb62-47ee-8cfc-733da84d7351" containerName="payload-pre-processing" containerID="cri-o://f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26" gracePeriod=25 Oct 30 14:49:24.957370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:24.957339 5824 generic.go:356] "Generic (PLEG): container finished" podID="7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f" containerID="73b758e562477aa3c447bb37deb3c66a8af01eb7c4a2dd45a082b8e2d1ae9107" exitCode=0 Oct 30 14:49:24.957724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:24.957423 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" event={"ID":"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f","Type":"ContainerDied","Data":"73b758e562477aa3c447bb37deb3c66a8af01eb7c4a2dd45a082b8e2d1ae9107"} Oct 30 14:49:25.162293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:25.162262 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl"] Oct 30 14:49:26.108330 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.108308 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:26.232904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.232841 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-kube-api-access-9scw5\" (UniqueName: \"kubernetes.io/projected/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-kube-api-access-9scw5\") pod \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " Oct 30 14:49:26.233032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.232936 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-openshift-ca\") pod \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " Oct 30 14:49:26.233032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.232985 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-xks-ca\") pod \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\" (UID: \"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f\") " Oct 30 14:49:26.233301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.233274 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-openshift-ca" pod "7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f" (UID: "7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:26.233499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.233485 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:26.234924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.234888 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-xks-ca" pod "7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f" (UID: "7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:26.235035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.234927 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-kube-api-access-9scw5" pod "7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f" (UID: "7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f"). InnerVolumeSpecName "kube-api-access-9scw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:26.334305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.334276 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9scw5\" (UniqueName: \"kubernetes.io/projected/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-kube-api-access-9scw5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:26.334305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.334300 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:26.783713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.783447 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" podUID="eaa3f864-0b20-41ca-9594-61803ab6411f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:26.783873 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.783703 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" podUID="f399c067-eb62-47ee-8cfc-733da84d7351" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:26.787346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.787319 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:49:26.824827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.824791 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:49:26.840631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.840600 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1711c3e-b183-4985-8d8a-7a4da293cf1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1711c3e-b183-4985-8d8a-7a4da293cf1d-plugins-config-volume\") pod \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\" (UID: \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\") " Oct 30 14:49:26.840857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.840834 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1711c3e-b183-4985-8d8a-7a4da293cf1d-kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/c1711c3e-b183-4985-8d8a-7a4da293cf1d-kube-api-access-slwl8\") pod \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\" (UID: \"c1711c3e-b183-4985-8d8a-7a4da293cf1d\") " Oct 30 14:49:26.841336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.841111 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1711c3e-b183-4985-8d8a-7a4da293cf1d-plugins-config-volume" pod "c1711c3e-b183-4985-8d8a-7a4da293cf1d" (UID: "c1711c3e-b183-4985-8d8a-7a4da293cf1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:26.841458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.841397 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1711c3e-b183-4985-8d8a-7a4da293cf1d-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:26.843262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.843237 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1711c3e-b183-4985-8d8a-7a4da293cf1d-kube-api-access-slwl8" pod "c1711c3e-b183-4985-8d8a-7a4da293cf1d" (UID: "c1711c3e-b183-4985-8d8a-7a4da293cf1d"). InnerVolumeSpecName "kube-api-access-slwl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:26.942033 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.941996 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b417b1-8f7a-421b-8db2-4fa9ac539d94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b417b1-8f7a-421b-8db2-4fa9ac539d94-plugins-config-volume\") pod \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\" (UID: \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\") " Oct 30 14:49:26.942405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.942359 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b417b1-8f7a-421b-8db2-4fa9ac539d94-kube-api-access-pccfl\" (UniqueName: \"kubernetes.io/projected/63b417b1-8f7a-421b-8db2-4fa9ac539d94-kube-api-access-pccfl\") pod \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\" (UID: \"63b417b1-8f7a-421b-8db2-4fa9ac539d94\") " Oct 30 14:49:26.942610 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.942582 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b417b1-8f7a-421b-8db2-4fa9ac539d94-plugins-config-volume" pod "63b417b1-8f7a-421b-8db2-4fa9ac539d94" (UID: "63b417b1-8f7a-421b-8db2-4fa9ac539d94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:26.943363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.943335 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwl8\" (UniqueName: \"kubernetes.io/projected/c1711c3e-b183-4985-8d8a-7a4da293cf1d-kube-api-access-slwl8\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:26.943363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.943364 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b417b1-8f7a-421b-8db2-4fa9ac539d94-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:26.945026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.944908 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b417b1-8f7a-421b-8db2-4fa9ac539d94-kube-api-access-pccfl" pod "63b417b1-8f7a-421b-8db2-4fa9ac539d94" (UID: "63b417b1-8f7a-421b-8db2-4fa9ac539d94"). InnerVolumeSpecName "kube-api-access-pccfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:26.973739 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.973710 5824 generic.go:356] "Generic (PLEG): container finished" podID="63b417b1-8f7a-421b-8db2-4fa9ac539d94" containerID="a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78" exitCode=0 Oct 30 14:49:26.973842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.973773 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" Oct 30 14:49:26.973842 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.973786 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" event={"ID":"63b417b1-8f7a-421b-8db2-4fa9ac539d94","Type":"ContainerDied","Data":"a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78"} Oct 30 14:49:26.973970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.973851 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh" event={"ID":"63b417b1-8f7a-421b-8db2-4fa9ac539d94","Type":"ContainerDied","Data":"45affa56099430479b0d02eaa813c0d9b54e7b34b6846ba53ddd7bab9a005701"} Oct 30 14:49:26.973970 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.973877 5824 scope.go:122] "RemoveContainer" containerID="a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78" Oct 30 14:49:26.975942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.975913 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" event={"ID":"7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f","Type":"ContainerDied","Data":"252e4535c445db0bc5943ced0bc1918bff552e13ed5c07c2e68fb4952eeeea44"} Oct 30 14:49:26.976253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.976232 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="252e4535c445db0bc5943ced0bc1918bff552e13ed5c07c2e68fb4952eeeea44" Oct 30 14:49:26.976341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.976268 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc" Oct 30 14:49:26.978428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.978404 5824 generic.go:356] "Generic (PLEG): container finished" podID="c1711c3e-b183-4985-8d8a-7a4da293cf1d" containerID="134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae" exitCode=0 Oct 30 14:49:26.978580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.978542 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" event={"ID":"c1711c3e-b183-4985-8d8a-7a4da293cf1d","Type":"ContainerDied","Data":"134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae"} Oct 30 14:49:26.978742 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.978610 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" event={"ID":"c1711c3e-b183-4985-8d8a-7a4da293cf1d","Type":"ContainerDied","Data":"67f2d2257af2127f0b1c10014029954f2edf7ca3cf8189f27c518ee32ec717bd"} Oct 30 14:49:26.978819 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.978755 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v" Oct 30 14:49:26.989437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.989419 5824 scope.go:122] "RemoveContainer" containerID="a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78" Oct 30 14:49:26.989781 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:26.989760 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78\": container with ID starting with a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78 not found: ID does not exist" containerID="a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78" Oct 30 14:49:26.989882 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.989786 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78"} err="failed to get container status \"a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78\": rpc error: code = NotFound desc = could not find container \"a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78\": container with ID starting with a84196b9efe2e5ea8865cae72f8b7122474b6433464c14c5e4ef0c9b21ed4f78 not found: ID does not exist" Oct 30 14:49:26.989882 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:26.989802 5824 scope.go:122] "RemoveContainer" containerID="134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae" Oct 30 14:49:27.000381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:27.000355 5824 scope.go:122] "RemoveContainer" containerID="134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae" Oct 30 14:49:27.000671 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:27.000646 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae\": container with ID starting with 134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae not found: ID does not exist" containerID="134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae" Oct 30 14:49:27.000733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:27.000678 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae"} err="failed to get container status \"134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae\": rpc error: code = NotFound desc = could not find container \"134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae\": container with ID starting with 134d0c386c493b6b77de31079b6928c2aaf16d45343cd56926bfec5be9cc98ae not found: ID does not exist" Oct 30 14:49:27.044601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:27.044538 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pccfl\" (UniqueName: \"kubernetes.io/projected/63b417b1-8f7a-421b-8db2-4fa9ac539d94-kube-api-access-pccfl\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:27.050692 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:27.050665 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh"] Oct 30 14:49:27.064263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:27.064237 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e371c3-67c8f8ffbc-8qcxh"] Oct 30 14:49:27.142180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:27.142147 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v"] Oct 30 14:49:27.148389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:27.148360 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e371c3-5b6c6695c6-hkq5v"] Oct 30 14:49:28.316844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.316813 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b417b1-8f7a-421b-8db2-4fa9ac539d94" path="/var/lib/kubelet/pods/63b417b1-8f7a-421b-8db2-4fa9ac539d94/volumes" Oct 30 14:49:28.317249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.317176 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1711c3e-b183-4985-8d8a-7a4da293cf1d" path="/var/lib/kubelet/pods/c1711c3e-b183-4985-8d8a-7a4da293cf1d/volumes" Oct 30 14:49:28.552077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.552043 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48"] Oct 30 14:49:28.552345 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.552315 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" podUID="5932998f-68b0-4495-8616-e45762f19574" containerName="maas-api" containerID="cri-o://17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34" gracePeriod=30 Oct 30 14:49:28.826486 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.826424 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:49:28.967395 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.967353 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-metrics-certs\") pod \"5932998f-68b0-4495-8616-e45762f19574\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " Oct 30 14:49:28.967589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.967403 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-maas-api-tls\") pod \"5932998f-68b0-4495-8616-e45762f19574\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " Oct 30 14:49:28.967664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.967591 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5932998f-68b0-4495-8616-e45762f19574-kube-api-access-kpdff\" (UniqueName: \"kubernetes.io/projected/5932998f-68b0-4495-8616-e45762f19574-kube-api-access-kpdff\") pod \"5932998f-68b0-4495-8616-e45762f19574\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " Oct 30 14:49:28.967820 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.967776 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5932998f-68b0-4495-8616-e45762f19574-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932998f-68b0-4495-8616-e45762f19574-openshift-service-ca\") pod \"5932998f-68b0-4495-8616-e45762f19574\" (UID: \"5932998f-68b0-4495-8616-e45762f19574\") " Oct 30 14:49:28.968319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.968288 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5932998f-68b0-4495-8616-e45762f19574-openshift-service-ca" pod "5932998f-68b0-4495-8616-e45762f19574" (UID: "5932998f-68b0-4495-8616-e45762f19574"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:28.968460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.968423 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5932998f-68b0-4495-8616-e45762f19574-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:28.969658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.969630 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-maas-api-tls" pod "5932998f-68b0-4495-8616-e45762f19574" (UID: "5932998f-68b0-4495-8616-e45762f19574"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:28.969771 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.969677 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-metrics-certs" pod "5932998f-68b0-4495-8616-e45762f19574" (UID: "5932998f-68b0-4495-8616-e45762f19574"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:28.969850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.969831 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5932998f-68b0-4495-8616-e45762f19574-kube-api-access-kpdff" pod "5932998f-68b0-4495-8616-e45762f19574" (UID: "5932998f-68b0-4495-8616-e45762f19574"). InnerVolumeSpecName "kube-api-access-kpdff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:29.010212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.999252 5824 generic.go:356] "Generic (PLEG): container finished" podID="5932998f-68b0-4495-8616-e45762f19574" containerID="17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34" exitCode=0 Oct 30 14:49:29.010212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.999365 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" event={"ID":"5932998f-68b0-4495-8616-e45762f19574","Type":"ContainerDied","Data":"17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34"} Oct 30 14:49:29.010212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.999419 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" event={"ID":"5932998f-68b0-4495-8616-e45762f19574","Type":"ContainerDied","Data":"37f276128561a3d15f2b26e23b51bc987261349506bf9da2ff7e3a665a216d95"} Oct 30 14:49:29.010212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.999453 5824 scope.go:122] "RemoveContainer" containerID="17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34" Oct 30 14:49:29.010212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:28.999612 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48" Oct 30 14:49:29.027474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:29.024822 5824 scope.go:122] "RemoveContainer" containerID="17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34" Oct 30 14:49:29.027474 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:29.025262 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34\": container with ID starting with 17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34 not found: ID does not exist" containerID="17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34" Oct 30 14:49:29.027474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:29.025292 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34"} err="failed to get container status \"17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34\": rpc error: code = NotFound desc = could not find container \"17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34\": container with ID starting with 17c3c6df8a858a9c98fd632b95766132417c3cfcec921a5597b34ab7cca17d34 not found: ID does not exist" Oct 30 14:49:29.070299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:29.070269 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:29.070499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:29.070480 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5932998f-68b0-4495-8616-e45762f19574-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:29.070499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:29.070502 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpdff\" (UniqueName: \"kubernetes.io/projected/5932998f-68b0-4495-8616-e45762f19574-kube-api-access-kpdff\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:29.073019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:29.072993 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48"] Oct 30 14:49:29.080757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:29.080704 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5704d6-786c79bd48-9sv48"] Oct 30 14:49:30.162559 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.162516 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" podUID="7d4d4407-499b-42c4-8a84-7a30c4b2f608" containerName="payload-processing" containerID="cri-o://d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76" gracePeriod=25 Oct 30 14:49:30.318728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.318695 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5932998f-68b0-4495-8616-e45762f19574" path="/var/lib/kubelet/pods/5932998f-68b0-4495-8616-e45762f19574/volumes" Oct 30 14:49:30.427597 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.427570 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:49:30.484947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.484919 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4d4407-499b-42c4-8a84-7a30c4b2f608-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4d4407-499b-42c4-8a84-7a30c4b2f608-plugins-config-volume\") pod \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\" (UID: \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\") " Oct 30 14:49:30.485082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.485062 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4d4407-499b-42c4-8a84-7a30c4b2f608-kube-api-access-2dtr6\" (UniqueName: \"kubernetes.io/projected/7d4d4407-499b-42c4-8a84-7a30c4b2f608-kube-api-access-2dtr6\") pod \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\" (UID: \"7d4d4407-499b-42c4-8a84-7a30c4b2f608\") " Oct 30 14:49:30.485346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.485320 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4d4407-499b-42c4-8a84-7a30c4b2f608-plugins-config-volume" pod "7d4d4407-499b-42c4-8a84-7a30c4b2f608" (UID: "7d4d4407-499b-42c4-8a84-7a30c4b2f608"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:30.487375 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.487350 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4d4407-499b-42c4-8a84-7a30c4b2f608-kube-api-access-2dtr6" pod "7d4d4407-499b-42c4-8a84-7a30c4b2f608" (UID: "7d4d4407-499b-42c4-8a84-7a30c4b2f608"). InnerVolumeSpecName "kube-api-access-2dtr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:30.586617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.586589 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtr6\" (UniqueName: \"kubernetes.io/projected/7d4d4407-499b-42c4-8a84-7a30c4b2f608-kube-api-access-2dtr6\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:30.586617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:30.586616 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4d4407-499b-42c4-8a84-7a30c4b2f608-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:31.013848 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.013813 5824 generic.go:356] "Generic (PLEG): container finished" podID="7d4d4407-499b-42c4-8a84-7a30c4b2f608" containerID="d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76" exitCode=0 Oct 30 14:49:31.014019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.013855 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" event={"ID":"7d4d4407-499b-42c4-8a84-7a30c4b2f608","Type":"ContainerDied","Data":"d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76"} Oct 30 14:49:31.014019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.013907 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" event={"ID":"7d4d4407-499b-42c4-8a84-7a30c4b2f608","Type":"ContainerDied","Data":"fcf016bccffdf24a3d426d6b5619015150f8ca976ff4899bf4caff128d0c4e94"} Oct 30 14:49:31.014019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.013927 5824 scope.go:122] "RemoveContainer" containerID="d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76" Oct 30 14:49:31.014019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.013861 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl" Oct 30 14:49:31.027946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.027927 5824 scope.go:122] "RemoveContainer" containerID="d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76" Oct 30 14:49:31.028288 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:31.028264 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76\": container with ID starting with d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76 not found: ID does not exist" containerID="d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76" Oct 30 14:49:31.028387 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.028298 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76"} err="failed to get container status \"d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76\": rpc error: code = NotFound desc = could not find container \"d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76\": container with ID starting with d18e988781b71eb61737dfb06fe65a6b39b215bbfc1208e493e080b8fc9dbc76 not found: ID does not exist" Oct 30 14:49:31.081199 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.081159 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl"] Oct 30 14:49:31.083840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.083816 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f1f206-5bc75d57d8-kb9kl"] Oct 30 14:49:31.678096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.678064 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4"] Oct 30 14:49:31.690392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:31.690365 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f1f206-vhgv4"] Oct 30 14:49:32.196305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:32.196272 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb"] Oct 30 14:49:32.211494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:32.211469 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-75443e-dcnwb"] Oct 30 14:49:32.317561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:32.317528 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653" path="/var/lib/kubelet/pods/0c2e078c-bc3b-41ea-8b90-7bf3b6c3e653/volumes" Oct 30 14:49:32.318075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:32.318054 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4d4407-499b-42c4-8a84-7a30c4b2f608" path="/var/lib/kubelet/pods/7d4d4407-499b-42c4-8a84-7a30c4b2f608/volumes" Oct 30 14:49:32.318510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:32.318495 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc14879-32af-4e94-bd53-7475533913d8" path="/var/lib/kubelet/pods/abc14879-32af-4e94-bd53-7475533913d8/volumes" Oct 30 14:49:33.252971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:33.252939 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn"] Oct 30 14:49:33.285829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:33.285804 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs"] Oct 30 14:49:33.643598 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:33.643498 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:49:33.786286 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:33.786253 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972"] Oct 30 14:49:33.786516 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:33.786485 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" podUID="4276ef28-e265-4ee5-9e4b-da715f993aaa" containerName="maas-api" containerID="cri-o://ec2120ab1545f223a4ed83e9ded93746b8826e32c8cf03b322896f0776b60ea7" gracePeriod=30 Oct 30 14:49:34.038786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.038756 5824 generic.go:356] "Generic (PLEG): container finished" podID="4276ef28-e265-4ee5-9e4b-da715f993aaa" containerID="ec2120ab1545f223a4ed83e9ded93746b8826e32c8cf03b322896f0776b60ea7" exitCode=0 Oct 30 14:49:34.038924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.038832 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" event={"ID":"4276ef28-e265-4ee5-9e4b-da715f993aaa","Type":"ContainerDied","Data":"ec2120ab1545f223a4ed83e9ded93746b8826e32c8cf03b322896f0776b60ea7"} Oct 30 14:49:34.038924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.038901 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" event={"ID":"4276ef28-e265-4ee5-9e4b-da715f993aaa","Type":"ContainerDied","Data":"3c65df5c0ca64a4102a25e4f15a73e0bd4ee79f8b2b84ca7c2880a96e3e8dd01"} Oct 30 14:49:34.038924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.038922 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c65df5c0ca64a4102a25e4f15a73e0bd4ee79f8b2b84ca7c2880a96e3e8dd01" Oct 30 14:49:34.065229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.062570 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:49:34.225656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.225618 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4276ef28-e265-4ee5-9e4b-da715f993aaa-kube-api-access-qvqw4\" (UniqueName: \"kubernetes.io/projected/4276ef28-e265-4ee5-9e4b-da715f993aaa-kube-api-access-qvqw4\") pod \"4276ef28-e265-4ee5-9e4b-da715f993aaa\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " Oct 30 14:49:34.225837 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.225805 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-metrics-certs\") pod \"4276ef28-e265-4ee5-9e4b-da715f993aaa\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " Oct 30 14:49:34.225901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.225845 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-maas-api-tls\") pod \"4276ef28-e265-4ee5-9e4b-da715f993aaa\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " Oct 30 14:49:34.225954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.225905 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4276ef28-e265-4ee5-9e4b-da715f993aaa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276ef28-e265-4ee5-9e4b-da715f993aaa-openshift-service-ca\") pod \"4276ef28-e265-4ee5-9e4b-da715f993aaa\" (UID: \"4276ef28-e265-4ee5-9e4b-da715f993aaa\") " Oct 30 14:49:34.226393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.226345 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4276ef28-e265-4ee5-9e4b-da715f993aaa-openshift-service-ca" pod "4276ef28-e265-4ee5-9e4b-da715f993aaa" (UID: "4276ef28-e265-4ee5-9e4b-da715f993aaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:34.226877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.226844 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276ef28-e265-4ee5-9e4b-da715f993aaa-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:34.228657 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.228603 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4276ef28-e265-4ee5-9e4b-da715f993aaa-kube-api-access-qvqw4" pod "4276ef28-e265-4ee5-9e4b-da715f993aaa" (UID: "4276ef28-e265-4ee5-9e4b-da715f993aaa"). InnerVolumeSpecName "kube-api-access-qvqw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:34.228772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.228740 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-metrics-certs" pod "4276ef28-e265-4ee5-9e4b-da715f993aaa" (UID: "4276ef28-e265-4ee5-9e4b-da715f993aaa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:34.228941 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.228918 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-maas-api-tls" pod "4276ef28-e265-4ee5-9e4b-da715f993aaa" (UID: "4276ef28-e265-4ee5-9e4b-da715f993aaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:34.327659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.327632 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:34.327659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.327658 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276ef28-e265-4ee5-9e4b-da715f993aaa-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:34.328028 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:34.327667 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvqw4\" (UniqueName: \"kubernetes.io/projected/4276ef28-e265-4ee5-9e4b-da715f993aaa-kube-api-access-qvqw4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:35.045095 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:35.045065 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972" Oct 30 14:49:35.103032 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:35.103001 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972"] Oct 30 14:49:35.116427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:35.116392 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a59ecb-667ff54d8f-fd972"] Oct 30 14:49:36.316521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:36.316489 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4276ef28-e265-4ee5-9e4b-da715f993aaa" path="/var/lib/kubelet/pods/4276ef28-e265-4ee5-9e4b-da715f993aaa/volumes" Oct 30 14:49:36.783753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:36.783711 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" podUID="eaa3f864-0b20-41ca-9594-61803ab6411f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:36.784145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:36.784122 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" podUID="f399c067-eb62-47ee-8cfc-733da84d7351" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:38.253479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:38.253441 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" podUID="99d95ba4-4fb9-46d2-8b43-08f95c4241bb" containerName="payload-processing" containerID="cri-o://4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94" gracePeriod=25 Oct 30 14:49:38.286909 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:38.286838 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" podUID="7e1277a3-8129-4d26-9e3e-612d4f7cb14a" containerName="payload-pre-processing" containerID="cri-o://660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f" gracePeriod=25 Oct 30 14:49:40.054000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.053965 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm"] Oct 30 14:49:40.054501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.054227 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" containerID="cri-o://58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee" gracePeriod=30 Oct 30 14:49:40.762422 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762392 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4"] Oct 30 14:49:40.762908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762894 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f" containerName="revoke-keys" Oct 30 14:49:40.762956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762909 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f" containerName="revoke-keys" Oct 30 14:49:40.762956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762952 5824 cpu_manager.go:414] "containerMap: removing container" podUID="63b417b1-8f7a-421b-8db2-4fa9ac539d94" containerName="payload-pre-processing" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762958 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b417b1-8f7a-421b-8db2-4fa9ac539d94" containerName="payload-pre-processing" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762967 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c1711c3e-b183-4985-8d8a-7a4da293cf1d" containerName="payload-processing" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762971 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1711c3e-b183-4985-8d8a-7a4da293cf1d" containerName="payload-processing" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762983 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5932998f-68b0-4495-8616-e45762f19574" containerName="maas-api" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762987 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5932998f-68b0-4495-8616-e45762f19574" containerName="maas-api" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.762995 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7d4d4407-499b-42c4-8a84-7a30c4b2f608" containerName="payload-processing" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763000 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4d4407-499b-42c4-8a84-7a30c4b2f608" containerName="payload-processing" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763005 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4276ef28-e265-4ee5-9e4b-da715f993aaa" containerName="maas-api" Oct 30 14:49:40.763025 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763010 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4276ef28-e265-4ee5-9e4b-da715f993aaa" containerName="maas-api" Oct 30 14:49:40.763306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763080 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b417b1-8f7a-421b-8db2-4fa9ac539d94" containerName="payload-pre-processing" Oct 30 14:49:40.763306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763087 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1711c3e-b183-4985-8d8a-7a4da293cf1d" containerName="payload-processing" Oct 30 14:49:40.763306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763099 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5932998f-68b0-4495-8616-e45762f19574" containerName="maas-api" Oct 30 14:49:40.763306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763108 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4d4407-499b-42c4-8a84-7a30c4b2f608" containerName="payload-processing" Oct 30 14:49:40.763306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763122 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4276ef28-e265-4ee5-9e4b-da715f993aaa" containerName="maas-api" Oct 30 14:49:40.763306 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.763132 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f" containerName="revoke-keys" Oct 30 14:49:40.767161 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.767135 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:40.786472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.786450 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4"] Oct 30 14:49:40.816517 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.816486 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:49:40.816644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.816533 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:49:40.889155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.889129 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:40.889303 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.889166 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvfn\" (UniqueName: \"kubernetes.io/projected/2d94f8ca-20da-4151-ab97-4ad52c787fb1-kube-api-access-jzvfn\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:40.889376 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.889339 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:40.990157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.990088 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:40.990157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.990127 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvfn\" (UniqueName: \"kubernetes.io/projected/2d94f8ca-20da-4151-ab97-4ad52c787fb1-kube-api-access-jzvfn\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:40.990354 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.990218 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:40.990858 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.990834 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:40.992399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:40.992379 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:41.016794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:41.016742 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvfn\" (UniqueName: \"kubernetes.io/projected/2d94f8ca-20da-4151-ab97-4ad52c787fb1-kube-api-access-jzvfn\") pod \"maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:41.079686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:41.079648 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:41.342225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:41.342126 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m"] Oct 30 14:49:41.342405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:41.342344 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" containerID="cri-o://5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11" gracePeriod=30 Oct 30 14:49:41.384499 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:49:41.384466 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d94f8ca_20da_4151_ab97_4ad52c787fb1.slice/crio-cf2d4e25e084fd6ba239c59547810b62c8abc7209d17af51544e06451a84f09a WatchSource:0}: Error finding container cf2d4e25e084fd6ba239c59547810b62c8abc7209d17af51544e06451a84f09a: Status 404 returned error can't find the container with id cf2d4e25e084fd6ba239c59547810b62c8abc7209d17af51544e06451a84f09a Oct 30 14:49:41.384809 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:41.384788 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4"] Oct 30 14:49:42.094264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:42.094225 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" event={"ID":"2d94f8ca-20da-4151-ab97-4ad52c787fb1","Type":"ContainerStarted","Data":"d7a3a59a225de34b3b5df735523ae7a6f105a71bb462e394b807d7142c141563"} Oct 30 14:49:42.094757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:42.094295 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" event={"ID":"2d94f8ca-20da-4151-ab97-4ad52c787fb1","Type":"ContainerStarted","Data":"cf2d4e25e084fd6ba239c59547810b62c8abc7209d17af51544e06451a84f09a"} Oct 30 14:49:43.102562 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:43.102528 5824 generic.go:356] "Generic (PLEG): container finished" podID="2d94f8ca-20da-4151-ab97-4ad52c787fb1" containerID="d7a3a59a225de34b3b5df735523ae7a6f105a71bb462e394b807d7142c141563" exitCode=0 Oct 30 14:49:43.102999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:43.102611 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" event={"ID":"2d94f8ca-20da-4151-ab97-4ad52c787fb1","Type":"ContainerDied","Data":"d7a3a59a225de34b3b5df735523ae7a6f105a71bb462e394b807d7142c141563"} Oct 30 14:49:44.247615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.247591 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:44.385901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.385830 5824 patch_prober.go:29] interesting pod/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:49:44.385901 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.385878 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:49:44.424362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.424337 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-xks-ca\") pod \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " Oct 30 14:49:44.424535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.424493 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d94f8ca-20da-4151-ab97-4ad52c787fb1-kube-api-access-jzvfn\" (UniqueName: \"kubernetes.io/projected/2d94f8ca-20da-4151-ab97-4ad52c787fb1-kube-api-access-jzvfn\") pod \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " Oct 30 14:49:44.424535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.424524 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-openshift-ca\") pod \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\" (UID: \"2d94f8ca-20da-4151-ab97-4ad52c787fb1\") " Oct 30 14:49:44.425018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.424986 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-openshift-ca" pod "2d94f8ca-20da-4151-ab97-4ad52c787fb1" (UID: "2d94f8ca-20da-4151-ab97-4ad52c787fb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:44.425253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.425230 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:44.426622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.426599 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-xks-ca" pod "2d94f8ca-20da-4151-ab97-4ad52c787fb1" (UID: "2d94f8ca-20da-4151-ab97-4ad52c787fb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:44.426744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.426708 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d94f8ca-20da-4151-ab97-4ad52c787fb1-kube-api-access-jzvfn" pod "2d94f8ca-20da-4151-ab97-4ad52c787fb1" (UID: "2d94f8ca-20da-4151-ab97-4ad52c787fb1"). InnerVolumeSpecName "kube-api-access-jzvfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:44.526305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.526279 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvfn\" (UniqueName: \"kubernetes.io/projected/2d94f8ca-20da-4151-ab97-4ad52c787fb1-kube-api-access-jzvfn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:44.526305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.526303 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94f8ca-20da-4151-ab97-4ad52c787fb1-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:44.774744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.774716 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:49:44.800899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.800878 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:49:44.930302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.930279 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f399c067-eb62-47ee-8cfc-733da84d7351-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399c067-eb62-47ee-8cfc-733da84d7351-plugins-config-volume\") pod \"f399c067-eb62-47ee-8cfc-733da84d7351\" (UID: \"f399c067-eb62-47ee-8cfc-733da84d7351\") " Oct 30 14:49:44.930479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.930330 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f399c067-eb62-47ee-8cfc-733da84d7351-kube-api-access-h8qpv\" (UniqueName: \"kubernetes.io/projected/f399c067-eb62-47ee-8cfc-733da84d7351-kube-api-access-h8qpv\") pod \"f399c067-eb62-47ee-8cfc-733da84d7351\" (UID: \"f399c067-eb62-47ee-8cfc-733da84d7351\") " Oct 30 14:49:44.930479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.930470 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa3f864-0b20-41ca-9594-61803ab6411f-kube-api-access-4dl95\" (UniqueName: \"kubernetes.io/projected/eaa3f864-0b20-41ca-9594-61803ab6411f-kube-api-access-4dl95\") pod \"eaa3f864-0b20-41ca-9594-61803ab6411f\" (UID: \"eaa3f864-0b20-41ca-9594-61803ab6411f\") " Oct 30 14:49:44.930592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.930574 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa3f864-0b20-41ca-9594-61803ab6411f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa3f864-0b20-41ca-9594-61803ab6411f-plugins-config-volume\") pod \"eaa3f864-0b20-41ca-9594-61803ab6411f\" (UID: \"eaa3f864-0b20-41ca-9594-61803ab6411f\") " Oct 30 14:49:44.930693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.930669 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f399c067-eb62-47ee-8cfc-733da84d7351-plugins-config-volume" pod "f399c067-eb62-47ee-8cfc-733da84d7351" (UID: "f399c067-eb62-47ee-8cfc-733da84d7351"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:44.930920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.930897 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa3f864-0b20-41ca-9594-61803ab6411f-plugins-config-volume" pod "eaa3f864-0b20-41ca-9594-61803ab6411f" (UID: "eaa3f864-0b20-41ca-9594-61803ab6411f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:44.931370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.931344 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399c067-eb62-47ee-8cfc-733da84d7351-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:44.931370 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.931362 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa3f864-0b20-41ca-9594-61803ab6411f-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:44.932709 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.932683 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f399c067-eb62-47ee-8cfc-733da84d7351-kube-api-access-h8qpv" pod "f399c067-eb62-47ee-8cfc-733da84d7351" (UID: "f399c067-eb62-47ee-8cfc-733da84d7351"). InnerVolumeSpecName "kube-api-access-h8qpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:44.932835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:44.932818 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa3f864-0b20-41ca-9594-61803ab6411f-kube-api-access-4dl95" pod "eaa3f864-0b20-41ca-9594-61803ab6411f" (UID: "eaa3f864-0b20-41ca-9594-61803ab6411f"). InnerVolumeSpecName "kube-api-access-4dl95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:45.032481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.032453 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8qpv\" (UniqueName: \"kubernetes.io/projected/f399c067-eb62-47ee-8cfc-733da84d7351-kube-api-access-h8qpv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.032481 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.032476 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dl95\" (UniqueName: \"kubernetes.io/projected/eaa3f864-0b20-41ca-9594-61803ab6411f-kube-api-access-4dl95\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.116539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.116513 5824 generic.go:356] "Generic (PLEG): container finished" podID="eaa3f864-0b20-41ca-9594-61803ab6411f" containerID="5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a" exitCode=0 Oct 30 14:49:45.116660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.116581 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" Oct 30 14:49:45.116660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.116604 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" event={"ID":"eaa3f864-0b20-41ca-9594-61803ab6411f","Type":"ContainerDied","Data":"5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a"} Oct 30 14:49:45.116774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.116665 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc" event={"ID":"eaa3f864-0b20-41ca-9594-61803ab6411f","Type":"ContainerDied","Data":"b62ec6a15450ebad16f0ebff05fec38bedff8537c57cd1175a9b25b75cfa3484"} Oct 30 14:49:45.116774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.116698 5824 scope.go:122] "RemoveContainer" containerID="5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a" Oct 30 14:49:45.118184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.118166 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" Oct 30 14:49:45.118288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.118198 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4" event={"ID":"2d94f8ca-20da-4151-ab97-4ad52c787fb1","Type":"ContainerDied","Data":"cf2d4e25e084fd6ba239c59547810b62c8abc7209d17af51544e06451a84f09a"} Oct 30 14:49:45.118288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.118242 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf2d4e25e084fd6ba239c59547810b62c8abc7209d17af51544e06451a84f09a" Oct 30 14:49:45.119845 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.119821 5824 generic.go:356] "Generic (PLEG): container finished" podID="f399c067-eb62-47ee-8cfc-733da84d7351" containerID="f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26" exitCode=0 Oct 30 14:49:45.119928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.119865 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" Oct 30 14:49:45.119928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.119894 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" event={"ID":"f399c067-eb62-47ee-8cfc-733da84d7351","Type":"ContainerDied","Data":"f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26"} Oct 30 14:49:45.120006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.119936 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp" event={"ID":"f399c067-eb62-47ee-8cfc-733da84d7351","Type":"ContainerDied","Data":"f151a7bd8495cb0825edc099017500014f3c29d12041ed5aa94a615e441b4e5a"} Oct 30 14:49:45.170800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.170780 5824 scope.go:122] "RemoveContainer" containerID="5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a" Oct 30 14:49:45.171091 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:45.171057 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a\": container with ID starting with 5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a not found: ID does not exist" containerID="5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a" Oct 30 14:49:45.171158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.171091 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a"} err="failed to get container status \"5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a\": rpc error: code = NotFound desc = could not find container \"5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a\": container with ID starting with 5ec897c11ff0639d849b96d4cd6f53da6fea1d6f0a1a83adb4815b662270aa9a not found: ID does not exist" Oct 30 14:49:45.171158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.171113 5824 scope.go:122] "RemoveContainer" containerID="f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26" Oct 30 14:49:45.181706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.181690 5824 scope.go:122] "RemoveContainer" containerID="f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26" Oct 30 14:49:45.182003 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:45.181981 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26\": container with ID starting with f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26 not found: ID does not exist" containerID="f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26" Oct 30 14:49:45.182093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.182007 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26"} err="failed to get container status \"f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26\": rpc error: code = NotFound desc = could not find container \"f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26\": container with ID starting with f59e07797e61e563ef1d4cda6ec9a8abe6af313443ad997cafe22ab9b683fa26 not found: ID does not exist" Oct 30 14:49:45.195413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.195389 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp"] Oct 30 14:49:45.204164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.204136 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5704d6-68bd9c5899-75mlp"] Oct 30 14:49:45.244844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.244815 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc"] Oct 30 14:49:45.247154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.247129 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5704d6-695bc45c86-nhwbc"] Oct 30 14:49:45.333965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.333944 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:49:45.435403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435343 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-envoy\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435446 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-certs\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435475 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-token\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-token\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435502 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-data\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435527 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-socket\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435559 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istiod-ca-cert\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435588 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-kube-api-access-jszf8\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-kube-api-access-jszf8\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435685 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-podinfo\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435722 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-credential-socket\") pod \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\" (UID: \"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7\") " Oct 30 14:49:45.435803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435788 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-certs" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:45.435948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435815 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-data" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:45.435948 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.435831 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-socket" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:45.436150 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.436114 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-credential-socket" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:45.436311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.436130 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istiod-ca-cert" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:45.436311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.436257 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.436311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.436278 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.436311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.436288 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.436311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.436297 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.436556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.436316 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.438019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.437995 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-envoy" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:45.438103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.438016 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-token" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:45.438241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.438222 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-podinfo" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:49:45.438335 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.438317 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-kube-api-access-jszf8" pod "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" (UID: "f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7"). InnerVolumeSpecName "kube-api-access-jszf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:45.537034 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.537011 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.537130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.537035 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jszf8\" (UniqueName: \"kubernetes.io/projected/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-kube-api-access-jszf8\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.537130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.537050 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.537130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.537063 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:45.551582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:45.551560 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h"] Oct 30 14:49:46.128471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.128441 5824 generic.go:356] "Generic (PLEG): container finished" podID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerID="58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee" exitCode=0 Oct 30 14:49:46.128645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.128493 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" Oct 30 14:49:46.128645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.128530 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" event={"ID":"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7","Type":"ContainerDied","Data":"58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee"} Oct 30 14:49:46.128645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.128600 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm" event={"ID":"f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7","Type":"ContainerDied","Data":"28b2112a0a6430172beb228c3e340fed56db1318e40a937b5fc990ed389fa1c9"} Oct 30 14:49:46.128645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.128637 5824 scope.go:122] "RemoveContainer" containerID="58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee" Oct 30 14:49:46.139292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.139277 5824 scope.go:122] "RemoveContainer" containerID="58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee" Oct 30 14:49:46.139539 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:46.139519 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee\": container with ID starting with 58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee not found: ID does not exist" containerID="58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee" Oct 30 14:49:46.139604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.139545 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee"} err="failed to get container status \"58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee\": rpc error: code = NotFound desc = could not find container \"58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee\": container with ID starting with 58ed7c5baeedb5e191d0bbf7e95cc6449958aff311dd1e8bc0f826ba055dcaee not found: ID does not exist" Oct 30 14:49:46.188277 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.188254 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm"] Oct 30 14:49:46.201834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.201809 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f1f206-openshift-default-d9b5df6f9-lgvxm"] Oct 30 14:49:46.317076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.317050 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa3f864-0b20-41ca-9594-61803ab6411f" path="/var/lib/kubelet/pods/eaa3f864-0b20-41ca-9594-61803ab6411f/volumes" Oct 30 14:49:46.317458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.317443 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f399c067-eb62-47ee-8cfc-733da84d7351" path="/var/lib/kubelet/pods/f399c067-eb62-47ee-8cfc-733da84d7351/volumes" Oct 30 14:49:46.317835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.317822 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" path="/var/lib/kubelet/pods/f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7/volumes" Oct 30 14:49:46.604609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.604586 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:49:46.648942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.648916 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-certs\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.648975 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-socket\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649026 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-envoy\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649076 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-credential-socket\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649124 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-podinfo\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649169 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-token\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-token\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649228 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-certs" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:46.649305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649242 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-socket" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:46.649305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649237 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-kube-api-access-5dqxd\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-kube-api-access-5dqxd\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649347 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-data\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649405 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd96b37-6ca8-4034-8c64-b99ab1652451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cd96b37-6ca8-4034-8c64-b99ab1652451-istiod-ca-cert\") pod \"8cd96b37-6ca8-4034-8c64-b99ab1652451\" (UID: \"8cd96b37-6ca8-4034-8c64-b99ab1652451\") " Oct 30 14:49:46.649532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649490 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-credential-socket" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:46.649691 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649602 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-data" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:46.649925 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649902 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:46.650011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649947 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:46.650011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649958 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:46.650011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649966 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:46.650011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.649903 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd96b37-6ca8-4034-8c64-b99ab1652451-istiod-ca-cert" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:46.651479 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.651459 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-envoy" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:46.651644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.651611 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-podinfo" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:49:46.651915 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.651896 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-token" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:46.651988 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.651958 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-kube-api-access-5dqxd" pod "8cd96b37-6ca8-4034-8c64-b99ab1652451" (UID: "8cd96b37-6ca8-4034-8c64-b99ab1652451"). InnerVolumeSpecName "kube-api-access-5dqxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:46.751369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.751305 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:46.751369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.751329 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:46.751369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.751337 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:46.751369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.751346 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dqxd\" (UniqueName: \"kubernetes.io/projected/8cd96b37-6ca8-4034-8c64-b99ab1652451-kube-api-access-5dqxd\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:46.751369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:46.751355 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cd96b37-6ca8-4034-8c64-b99ab1652451-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:47.138138 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.138046 5824 generic.go:356] "Generic (PLEG): container finished" podID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerID="5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11" exitCode=0 Oct 30 14:49:47.138138 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.138094 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" Oct 30 14:49:47.138394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.138137 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" event={"ID":"8cd96b37-6ca8-4034-8c64-b99ab1652451","Type":"ContainerDied","Data":"5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11"} Oct 30 14:49:47.138394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.138211 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m" event={"ID":"8cd96b37-6ca8-4034-8c64-b99ab1652451","Type":"ContainerDied","Data":"eb9b36dba9c7e76179b0565a9a02e1f3ea2186141dc48af79842177c270d8521"} Oct 30 14:49:47.138394 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.138241 5824 scope.go:122] "RemoveContainer" containerID="5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11" Oct 30 14:49:47.150092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.150071 5824 scope.go:122] "RemoveContainer" containerID="5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11" Oct 30 14:49:47.150388 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:47.150365 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11\": container with ID starting with 5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11 not found: ID does not exist" containerID="5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11" Oct 30 14:49:47.150468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.150396 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11"} err="failed to get container status \"5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11\": rpc error: code = NotFound desc = could not find container \"5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11\": container with ID starting with 5f4391794642a582e76af4525dbfd6d228c935b61f83390b04310582a7c96b11 not found: ID does not exist" Oct 30 14:49:47.199799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.199765 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m"] Oct 30 14:49:47.212141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.212106 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-75443e-openshift-default-567ff8b6bc-22r8m"] Oct 30 14:49:47.617266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617227 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66"] Oct 30 14:49:47.617736 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617721 5824 cpu_manager.go:414] "containerMap: removing container" podUID="2d94f8ca-20da-4151-ab97-4ad52c787fb1" containerName="revoke-keys" Oct 30 14:49:47.617799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617736 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d94f8ca-20da-4151-ab97-4ad52c787fb1" containerName="revoke-keys" Oct 30 14:49:47.617799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617749 5824 cpu_manager.go:414] "containerMap: removing container" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" Oct 30 14:49:47.617799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617756 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" Oct 30 14:49:47.617799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617767 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" Oct 30 14:49:47.617799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617773 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" Oct 30 14:49:47.617799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617788 5824 cpu_manager.go:414] "containerMap: removing container" podUID="eaa3f864-0b20-41ca-9594-61803ab6411f" containerName="payload-processing" Oct 30 14:49:47.617799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617793 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa3f864-0b20-41ca-9594-61803ab6411f" containerName="payload-processing" Oct 30 14:49:47.618114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617815 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f399c067-eb62-47ee-8cfc-733da84d7351" containerName="payload-pre-processing" Oct 30 14:49:47.618114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617821 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f399c067-eb62-47ee-8cfc-733da84d7351" containerName="payload-pre-processing" Oct 30 14:49:47.618114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617918 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d94f8ca-20da-4151-ab97-4ad52c787fb1" containerName="revoke-keys" Oct 30 14:49:47.618114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617924 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" containerName="istio-proxy" Oct 30 14:49:47.618114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617933 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b77199-8c07-4f6a-98ba-5f68bf7d5ae7" containerName="istio-proxy" Oct 30 14:49:47.618114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617941 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa3f864-0b20-41ca-9594-61803ab6411f" containerName="payload-processing" Oct 30 14:49:47.618114 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.617949 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f399c067-eb62-47ee-8cfc-733da84d7351" containerName="payload-pre-processing" Oct 30 14:49:47.622910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.622888 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.647271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.647236 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:49:47.648390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.648366 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:49:47.657552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.657521 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66"] Oct 30 14:49:47.669838 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.669815 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzf4k\" (UniqueName: \"kubernetes.io/projected/31a94829-f193-41f6-9cc2-08449ca96d89-kube-api-access-vzf4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.669952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.669911 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.670042 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.669969 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.770781 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.770741 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzf4k\" (UniqueName: \"kubernetes.io/projected/31a94829-f193-41f6-9cc2-08449ca96d89-kube-api-access-vzf4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.770958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.770827 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.770958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.770872 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.771566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.771539 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.773013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.772990 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.796622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.796600 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzf4k\" (UniqueName: \"kubernetes.io/projected/31a94829-f193-41f6-9cc2-08449ca96d89-kube-api-access-vzf4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:47.898284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.898220 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" podUID="99d95ba4-4fb9-46d2-8b43-08f95c4241bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:47.898404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.898216 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" podUID="7e1277a3-8129-4d26-9e3e-612d4f7cb14a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:47.947803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:47.947780 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:48.249363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:48.249337 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66"] Oct 30 14:49:48.249548 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:49:48.249524 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a94829_f193_41f6_9cc2_08449ca96d89.slice/crio-fc6088378a165eaa4df5d47bf682e5bd76872810bd27390da3f6a14b27a2e137 WatchSource:0}: Error finding container fc6088378a165eaa4df5d47bf682e5bd76872810bd27390da3f6a14b27a2e137: Status 404 returned error can't find the container with id fc6088378a165eaa4df5d47bf682e5bd76872810bd27390da3f6a14b27a2e137 Oct 30 14:49:48.315951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:48.315927 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd96b37-6ca8-4034-8c64-b99ab1652451" path="/var/lib/kubelet/pods/8cd96b37-6ca8-4034-8c64-b99ab1652451/volumes" Oct 30 14:49:48.761897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:48.761868 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc"] Oct 30 14:49:48.765828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:48.765801 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a59ecb-ks9jc"] Oct 30 14:49:49.155973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:49.155941 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" event={"ID":"31a94829-f193-41f6-9cc2-08449ca96d89","Type":"ContainerStarted","Data":"1e5e5a6abd5a248977fab99fc6f7fdc987634fb23c4e7cf337ae9a595e101e83"} Oct 30 14:49:49.156077 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:49.156000 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" event={"ID":"31a94829-f193-41f6-9cc2-08449ca96d89","Type":"ContainerStarted","Data":"fc6088378a165eaa4df5d47bf682e5bd76872810bd27390da3f6a14b27a2e137"} Oct 30 14:49:50.164450 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.164407 5824 generic.go:356] "Generic (PLEG): container finished" podID="31a94829-f193-41f6-9cc2-08449ca96d89" containerID="1e5e5a6abd5a248977fab99fc6f7fdc987634fb23c4e7cf337ae9a595e101e83" exitCode=0 Oct 30 14:49:50.164865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.164499 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" event={"ID":"31a94829-f193-41f6-9cc2-08449ca96d89","Type":"ContainerDied","Data":"1e5e5a6abd5a248977fab99fc6f7fdc987634fb23c4e7cf337ae9a595e101e83"} Oct 30 14:49:50.298796 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.298772 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:50.316556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.316531 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f" path="/var/lib/kubelet/pods/7e4829b5-2fc2-4337-9d3a-8fa8dc8a6e3f/volumes" Oct 30 14:49:50.392999 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.392970 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-openshift-ca\") pod \"31a94829-f193-41f6-9cc2-08449ca96d89\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " Oct 30 14:49:50.393172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.393005 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a94829-f193-41f6-9cc2-08449ca96d89-kube-api-access-vzf4k\" (UniqueName: \"kubernetes.io/projected/31a94829-f193-41f6-9cc2-08449ca96d89-kube-api-access-vzf4k\") pod \"31a94829-f193-41f6-9cc2-08449ca96d89\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " Oct 30 14:49:50.393172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.393063 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-xks-ca\") pod \"31a94829-f193-41f6-9cc2-08449ca96d89\" (UID: \"31a94829-f193-41f6-9cc2-08449ca96d89\") " Oct 30 14:49:50.393438 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.393415 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-openshift-ca" pod "31a94829-f193-41f6-9cc2-08449ca96d89" (UID: "31a94829-f193-41f6-9cc2-08449ca96d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:50.393587 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.393570 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:50.395062 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.395041 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a94829-f193-41f6-9cc2-08449ca96d89-kube-api-access-vzf4k" pod "31a94829-f193-41f6-9cc2-08449ca96d89" (UID: "31a94829-f193-41f6-9cc2-08449ca96d89"). InnerVolumeSpecName "kube-api-access-vzf4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:50.395239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.395220 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-xks-ca" pod "31a94829-f193-41f6-9cc2-08449ca96d89" (UID: "31a94829-f193-41f6-9cc2-08449ca96d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:50.494605 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.494578 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzf4k\" (UniqueName: \"kubernetes.io/projected/31a94829-f193-41f6-9cc2-08449ca96d89-kube-api-access-vzf4k\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:50.494712 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:50.494610 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31a94829-f193-41f6-9cc2-08449ca96d89-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:51.180128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:51.180095 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" Oct 30 14:49:51.180128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:51.180110 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66" event={"ID":"31a94829-f193-41f6-9cc2-08449ca96d89","Type":"ContainerDied","Data":"fc6088378a165eaa4df5d47bf682e5bd76872810bd27390da3f6a14b27a2e137"} Oct 30 14:49:51.180550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:51.180198 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6088378a165eaa4df5d47bf682e5bd76872810bd27390da3f6a14b27a2e137" Oct 30 14:49:51.221600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:51.221575 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49"] Oct 30 14:49:52.345353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:52.345318 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66"] Oct 30 14:49:52.377709 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:52.377681 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2"] Oct 30 14:49:56.222656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:56.222622 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" podUID="650319fd-aeda-4ba4-9594-51ef46a181a9" containerName="payload-processing" containerID="cri-o://a87132cdbc0e64649aebf220ba321cfdb637e8976020fb3780651545b426ff68" gracePeriod=25 Oct 30 14:49:57.232840 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.232807 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" podUID="650319fd-aeda-4ba4-9594-51ef46a181a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:57.346352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.346322 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" podUID="7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" containerName="payload-processing" containerID="cri-o://aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b" gracePeriod=25 Oct 30 14:49:57.378443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.378385 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" podUID="ae95a746-48ba-4686-8f37-ae30206dc275" containerName="payload-pre-processing" containerID="cri-o://0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5" gracePeriod=25 Oct 30 14:49:57.565471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.565401 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg"] Oct 30 14:49:57.565933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.565917 5824 cpu_manager.go:414] "containerMap: removing container" podUID="31a94829-f193-41f6-9cc2-08449ca96d89" containerName="revoke-keys" Oct 30 14:49:57.565980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.565937 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a94829-f193-41f6-9cc2-08449ca96d89" containerName="revoke-keys" Oct 30 14:49:57.566066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.566057 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a94829-f193-41f6-9cc2-08449ca96d89" containerName="revoke-keys" Oct 30 14:49:57.569673 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.569655 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.588846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.588825 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg"] Oct 30 14:49:57.613965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.613938 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:49:57.614130 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.613979 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:49:57.665938 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.665912 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphdp\" (UniqueName: \"kubernetes.io/projected/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-kube-api-access-pphdp\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.666076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.665987 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.666122 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.666076 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.767066 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.767036 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphdp\" (UniqueName: \"kubernetes.io/projected/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-kube-api-access-pphdp\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.767252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.767137 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.767252 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.767176 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.767963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.767941 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.769985 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.769968 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.794034 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.794009 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphdp\" (UniqueName: \"kubernetes.io/projected/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-kube-api-access-pphdp\") pod \"maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.882949 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.882889 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:49:57.898096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.898068 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" podUID="7e1277a3-8129-4d26-9e3e-612d4f7cb14a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:57.898233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.898108 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" podUID="99d95ba4-4fb9-46d2-8b43-08f95c4241bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:49:57.986830 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:57.986797 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw"] Oct 30 14:49:58.025778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.025750 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7"] Oct 30 14:49:58.150462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.150386 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr"] Oct 30 14:49:58.156456 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.156427 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.177229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.173403 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr"] Oct 30 14:49:58.193724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.193695 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg"] Oct 30 14:49:58.203769 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:49:58.203695 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a41048c_1202_4c97_b5a4_8a3f2c1e76bb.slice/crio-6041ff09ad7a73209570fb7e92b68a89a52249b586add43451cfc1ab698c7bcf WatchSource:0}: Error finding container 6041ff09ad7a73209570fb7e92b68a89a52249b586add43451cfc1ab698c7bcf: Status 404 returned error can't find the container with id 6041ff09ad7a73209570fb7e92b68a89a52249b586add43451cfc1ab698c7bcf Oct 30 14:49:58.224991 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.224967 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" event={"ID":"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb","Type":"ContainerStarted","Data":"6041ff09ad7a73209570fb7e92b68a89a52249b586add43451cfc1ab698c7bcf"} Oct 30 14:49:58.273386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.273360 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.273704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.273404 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjmv\" (UniqueName: \"kubernetes.io/projected/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-kube-api-access-cbjmv\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.273704 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.273555 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.374893 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.374865 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.375046 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.374918 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjmv\" (UniqueName: \"kubernetes.io/projected/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-kube-api-access-cbjmv\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.375284 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.375256 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.375903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.375879 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.377302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.377281 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.401390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.401341 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjmv\" (UniqueName: \"kubernetes.io/projected/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-kube-api-access-cbjmv\") pod \"maas-api-revoke-sub-5f8ea5ce-9clsr\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.474891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.474304 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:49:58.531391 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.531361 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:49:58.580513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.579488 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-kube-api-access-m96hj\" (UniqueName: \"kubernetes.io/projected/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-kube-api-access-m96hj\") pod \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\" (UID: \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\") " Oct 30 14:49:58.580513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.579643 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-plugins-config-volume\") pod \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\" (UID: \"99d95ba4-4fb9-46d2-8b43-08f95c4241bb\") " Oct 30 14:49:58.580807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.580551 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-plugins-config-volume" pod "99d95ba4-4fb9-46d2-8b43-08f95c4241bb" (UID: "99d95ba4-4fb9-46d2-8b43-08f95c4241bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:58.587681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.587656 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-kube-api-access-m96hj" pod "99d95ba4-4fb9-46d2-8b43-08f95c4241bb" (UID: "99d95ba4-4fb9-46d2-8b43-08f95c4241bb"). InnerVolumeSpecName "kube-api-access-m96hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:58.592600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.592577 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:49:58.681328 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.681298 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-plugins-config-volume\") pod \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\" (UID: \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\") " Oct 30 14:49:58.681521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.681443 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-kube-api-access-4j8lh\" (UniqueName: \"kubernetes.io/projected/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-kube-api-access-4j8lh\") pod \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\" (UID: \"7e1277a3-8129-4d26-9e3e-612d4f7cb14a\") " Oct 30 14:49:58.681889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.681871 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m96hj\" (UniqueName: \"kubernetes.io/projected/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-kube-api-access-m96hj\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:58.681997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.681893 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d95ba4-4fb9-46d2-8b43-08f95c4241bb-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:58.681997 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.681895 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-plugins-config-volume" pod "7e1277a3-8129-4d26-9e3e-612d4f7cb14a" (UID: "7e1277a3-8129-4d26-9e3e-612d4f7cb14a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:58.683846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.683826 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-kube-api-access-4j8lh" pod "7e1277a3-8129-4d26-9e3e-612d4f7cb14a" (UID: "7e1277a3-8129-4d26-9e3e-612d4f7cb14a"). InnerVolumeSpecName "kube-api-access-4j8lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:58.783148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.783120 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:58.783264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.783152 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j8lh\" (UniqueName: \"kubernetes.io/projected/7e1277a3-8129-4d26-9e3e-612d4f7cb14a-kube-api-access-4j8lh\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:49:58.791617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:58.790623 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr"] Oct 30 14:49:58.795027 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:49:58.794782 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf78b24_b4b5_4c3d_a8e0_8d669aab1c81.slice/crio-4017dcf4181f6ae6b527900996638a71a521042d38b9875a727e2a398c4b7287 WatchSource:0}: Error finding container 4017dcf4181f6ae6b527900996638a71a521042d38b9875a727e2a398c4b7287: Status 404 returned error can't find the container with id 4017dcf4181f6ae6b527900996638a71a521042d38b9875a727e2a398c4b7287 Oct 30 14:49:59.233681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.233604 5824 generic.go:356] "Generic (PLEG): container finished" podID="8a41048c-1202-4c97-b5a4-8a3f2c1e76bb" containerID="31c5eec888fba258f11b264e313bd59aecd978076f7fef4664b286ed73769d38" exitCode=0 Oct 30 14:49:59.233811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.233694 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" event={"ID":"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb","Type":"ContainerDied","Data":"31c5eec888fba258f11b264e313bd59aecd978076f7fef4664b286ed73769d38"} Oct 30 14:49:59.235020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.234999 5824 generic.go:356] "Generic (PLEG): container finished" podID="7e1277a3-8129-4d26-9e3e-612d4f7cb14a" containerID="660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f" exitCode=0 Oct 30 14:49:59.235159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.235038 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" Oct 30 14:49:59.235159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.235085 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" event={"ID":"7e1277a3-8129-4d26-9e3e-612d4f7cb14a","Type":"ContainerDied","Data":"660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f"} Oct 30 14:49:59.235159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.235138 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs" event={"ID":"7e1277a3-8129-4d26-9e3e-612d4f7cb14a","Type":"ContainerDied","Data":"38d2f79151ee87421d258e3c148e9187bf60eb4134d18f0bced5d98b213627f5"} Oct 30 14:49:59.235327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.235170 5824 scope.go:122] "RemoveContainer" containerID="660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f" Oct 30 14:49:59.236917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.236891 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" event={"ID":"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81","Type":"ContainerStarted","Data":"29acc643cd8a82fb8b845b28fb6541e37c34a60f8665e4de4deb2af94f04bb8b"} Oct 30 14:49:59.237045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.236940 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" event={"ID":"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81","Type":"ContainerStarted","Data":"4017dcf4181f6ae6b527900996638a71a521042d38b9875a727e2a398c4b7287"} Oct 30 14:49:59.238432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.238410 5824 generic.go:356] "Generic (PLEG): container finished" podID="99d95ba4-4fb9-46d2-8b43-08f95c4241bb" containerID="4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94" exitCode=0 Oct 30 14:49:59.238545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.238466 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" Oct 30 14:49:59.238545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.238512 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" event={"ID":"99d95ba4-4fb9-46d2-8b43-08f95c4241bb","Type":"ContainerDied","Data":"4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94"} Oct 30 14:49:59.238634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.238547 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn" event={"ID":"99d95ba4-4fb9-46d2-8b43-08f95c4241bb","Type":"ContainerDied","Data":"78419e7640c9f3a373b43e2192e8b277b4ceed7e6145a6e8b650e7d86e1bc664"} Oct 30 14:49:59.246064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.246048 5824 scope.go:122] "RemoveContainer" containerID="660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f" Oct 30 14:49:59.246379 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:59.246359 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f\": container with ID starting with 660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f not found: ID does not exist" containerID="660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f" Oct 30 14:49:59.246442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.246391 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f"} err="failed to get container status \"660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f\": rpc error: code = NotFound desc = could not find container \"660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f\": container with ID starting with 660217d614d3d4e0d50c10ae49aed2c2c2b616bfa7f60b9ec241d9e0e674a09f not found: ID does not exist" Oct 30 14:49:59.246442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.246415 5824 scope.go:122] "RemoveContainer" containerID="4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94" Oct 30 14:49:59.255521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.255506 5824 scope.go:122] "RemoveContainer" containerID="4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94" Oct 30 14:49:59.255765 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:49:59.255747 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94\": container with ID starting with 4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94 not found: ID does not exist" containerID="4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94" Oct 30 14:49:59.255815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.255770 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94"} err="failed to get container status \"4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94\": rpc error: code = NotFound desc = could not find container \"4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94\": container with ID starting with 4aafffcc6ddae81116d77dde636a09bf7b5f5e521e80efc9caaac7b7708d9f94 not found: ID does not exist" Oct 30 14:49:59.329327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.329304 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs"] Oct 30 14:49:59.331437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.331417 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a59ecb-dfbdf6cdc-zsbgs"] Oct 30 14:49:59.368882 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.368859 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn"] Oct 30 14:49:59.375861 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.375842 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a59ecb-99d68b8b4-zwhsn"] Oct 30 14:49:59.417702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:49:59.417657 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" podStartSLOduration=1.417646355 podStartE2EDuration="1.417646355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:49:59.41596137 +0000 UTC m=+3629.216466813" watchObservedRunningTime="2026-10-30 14:49:59.417646355 +0000 UTC m=+3629.218151796" Oct 30 14:50:00.250757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.250722 5824 generic.go:356] "Generic (PLEG): container finished" podID="2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81" containerID="29acc643cd8a82fb8b845b28fb6541e37c34a60f8665e4de4deb2af94f04bb8b" exitCode=0 Oct 30 14:50:00.250945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.250803 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" event={"ID":"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81","Type":"ContainerDied","Data":"29acc643cd8a82fb8b845b28fb6541e37c34a60f8665e4de4deb2af94f04bb8b"} Oct 30 14:50:00.330643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.330614 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1277a3-8129-4d26-9e3e-612d4f7cb14a" path="/var/lib/kubelet/pods/7e1277a3-8129-4d26-9e3e-612d4f7cb14a/volumes" Oct 30 14:50:00.331238 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.331217 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d95ba4-4fb9-46d2-8b43-08f95c4241bb" path="/var/lib/kubelet/pods/99d95ba4-4fb9-46d2-8b43-08f95c4241bb/volumes" Oct 30 14:50:00.399825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.399801 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:50:00.500164 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.500136 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-kube-api-access-pphdp\" (UniqueName: \"kubernetes.io/projected/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-kube-api-access-pphdp\") pod \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " Oct 30 14:50:00.500337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.500182 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-xks-ca\") pod \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " Oct 30 14:50:00.500337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.500321 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-openshift-ca\") pod \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\" (UID: \"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb\") " Oct 30 14:50:00.500661 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.500634 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-openshift-ca" pod "8a41048c-1202-4c97-b5a4-8a3f2c1e76bb" (UID: "8a41048c-1202-4c97-b5a4-8a3f2c1e76bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:00.500878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.500831 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:00.502494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.502468 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-kube-api-access-pphdp" pod "8a41048c-1202-4c97-b5a4-8a3f2c1e76bb" (UID: "8a41048c-1202-4c97-b5a4-8a3f2c1e76bb"). InnerVolumeSpecName "kube-api-access-pphdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:00.502614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.502593 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-xks-ca" pod "8a41048c-1202-4c97-b5a4-8a3f2c1e76bb" (UID: "8a41048c-1202-4c97-b5a4-8a3f2c1e76bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:00.572890 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.572840 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" containerID="cri-o://1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa" gracePeriod=45 Oct 30 14:50:00.602019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.601982 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pphdp\" (UniqueName: \"kubernetes.io/projected/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-kube-api-access-pphdp\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:00.602131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.602022 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:00.831711 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.831676 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:50:00.905313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.905279 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489d0e4e-50f7-48c7-8df9-7482a3455878-kube-api-access-r2cp8\" (UniqueName: \"kubernetes.io/projected/489d0e4e-50f7-48c7-8df9-7482a3455878-kube-api-access-r2cp8\") pod \"489d0e4e-50f7-48c7-8df9-7482a3455878\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " Oct 30 14:50:00.905506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.905377 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/489d0e4e-50f7-48c7-8df9-7482a3455878-tls-certs\" (UniqueName: \"kubernetes.io/secret/489d0e4e-50f7-48c7-8df9-7482a3455878-tls-certs\") pod \"489d0e4e-50f7-48c7-8df9-7482a3455878\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " Oct 30 14:50:00.905579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.905541 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-home\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-home\") pod \"489d0e4e-50f7-48c7-8df9-7482a3455878\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " Oct 30 14:50:00.905644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.905600 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-tmp-dir\") pod \"489d0e4e-50f7-48c7-8df9-7482a3455878\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " Oct 30 14:50:00.905708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.905665 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-model-cache\") pod \"489d0e4e-50f7-48c7-8df9-7482a3455878\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " Oct 30 14:50:00.905708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.905688 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-dshm\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-dshm\") pod \"489d0e4e-50f7-48c7-8df9-7482a3455878\" (UID: \"489d0e4e-50f7-48c7-8df9-7482a3455878\") " Oct 30 14:50:00.905940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.905914 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-tmp-dir" pod "489d0e4e-50f7-48c7-8df9-7482a3455878" (UID: "489d0e4e-50f7-48c7-8df9-7482a3455878"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:00.906314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.906284 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:00.906424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.906382 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-home" pod "489d0e4e-50f7-48c7-8df9-7482a3455878" (UID: "489d0e4e-50f7-48c7-8df9-7482a3455878"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:00.906506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.906443 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-model-cache" pod "489d0e4e-50f7-48c7-8df9-7482a3455878" (UID: "489d0e4e-50f7-48c7-8df9-7482a3455878"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:00.911576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.908669 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489d0e4e-50f7-48c7-8df9-7482a3455878-tls-certs" pod "489d0e4e-50f7-48c7-8df9-7482a3455878" (UID: "489d0e4e-50f7-48c7-8df9-7482a3455878"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:00.914384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.914341 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-dshm" pod "489d0e4e-50f7-48c7-8df9-7482a3455878" (UID: "489d0e4e-50f7-48c7-8df9-7482a3455878"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:00.914384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.914358 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489d0e4e-50f7-48c7-8df9-7482a3455878-kube-api-access-r2cp8" pod "489d0e4e-50f7-48c7-8df9-7482a3455878" (UID: "489d0e4e-50f7-48c7-8df9-7482a3455878"). InnerVolumeSpecName "kube-api-access-r2cp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:00.967314 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.967278 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx"] Oct 30 14:50:00.967558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:00.967532 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" podUID="1d8724ac-5136-4a32-b9ab-7f0aa1277938" containerName="maas-api" containerID="cri-o://fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728" gracePeriod=30 Oct 30 14:50:01.007296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.007223 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.007296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.007258 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.007296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.007274 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2cp8\" (UniqueName: \"kubernetes.io/projected/489d0e4e-50f7-48c7-8df9-7482a3455878-kube-api-access-r2cp8\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.007296 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.007290 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/489d0e4e-50f7-48c7-8df9-7482a3455878-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.007633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.007310 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/489d0e4e-50f7-48c7-8df9-7482a3455878-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.210603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.210579 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:50:01.260613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.260537 5824 generic.go:356] "Generic (PLEG): container finished" podID="1d8724ac-5136-4a32-b9ab-7f0aa1277938" containerID="fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728" exitCode=0 Oct 30 14:50:01.260613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.260582 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" Oct 30 14:50:01.260836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.260633 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" event={"ID":"1d8724ac-5136-4a32-b9ab-7f0aa1277938","Type":"ContainerDied","Data":"fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728"} Oct 30 14:50:01.260836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.260695 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx" event={"ID":"1d8724ac-5136-4a32-b9ab-7f0aa1277938","Type":"ContainerDied","Data":"5642eab4e5762d8ca8af279750d746596eaa915c404d9a1faf258e261a9a3168"} Oct 30 14:50:01.260836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.260728 5824 scope.go:122] "RemoveContainer" containerID="fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728" Oct 30 14:50:01.262175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.262137 5824 generic.go:356] "Generic (PLEG): container finished" podID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerID="1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa" exitCode=0 Oct 30 14:50:01.262287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.262254 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" event={"ID":"489d0e4e-50f7-48c7-8df9-7482a3455878","Type":"ContainerDied","Data":"1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa"} Oct 30 14:50:01.262287 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.262278 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" event={"ID":"489d0e4e-50f7-48c7-8df9-7482a3455878","Type":"ContainerDied","Data":"802f5683b59d65d28f4ffc325193b501ea0e01dc60b7a5f30a0aa606150d0c82"} Oct 30 14:50:01.262382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.262321 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h" Oct 30 14:50:01.263980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.263961 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" Oct 30 14:50:01.264113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.264011 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg" event={"ID":"8a41048c-1202-4c97-b5a4-8a3f2c1e76bb","Type":"ContainerDied","Data":"6041ff09ad7a73209570fb7e92b68a89a52249b586add43451cfc1ab698c7bcf"} Oct 30 14:50:01.264113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.264066 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6041ff09ad7a73209570fb7e92b68a89a52249b586add43451cfc1ab698c7bcf" Oct 30 14:50:01.271614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.271566 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66"] Oct 30 14:50:01.275040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.275017 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-228454-h6p66"] Oct 30 14:50:01.276527 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.276508 5824 scope.go:122] "RemoveContainer" containerID="fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728" Oct 30 14:50:01.276848 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:01.276823 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728\": container with ID starting with fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728 not found: ID does not exist" containerID="fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728" Oct 30 14:50:01.276918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.276859 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728"} err="failed to get container status \"fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728\": rpc error: code = NotFound desc = could not find container \"fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728\": container with ID starting with fd01053a7d7443659c0792b8657a81bbd6fb58d4a860a1122f2a9b80543c5728 not found: ID does not exist" Oct 30 14:50:01.276918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.276881 5824 scope.go:122] "RemoveContainer" containerID="1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa" Oct 30 14:50:01.290660 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.290638 5824 scope.go:122] "RemoveContainer" containerID="1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa" Oct 30 14:50:01.291021 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:01.291002 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa\": container with ID starting with 1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa not found: ID does not exist" containerID="1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa" Oct 30 14:50:01.291095 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.291031 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa"} err="failed to get container status \"1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa\": rpc error: code = NotFound desc = could not find container \"1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa\": container with ID starting with 1e5ca924195c978b2ca078c795a52af40a9250a198b0afdf7824185e040aceaa not found: ID does not exist" Oct 30 14:50:01.311065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.310496 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-metrics-certs\") pod \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " Oct 30 14:50:01.311065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.310548 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-maas-api-tls\") pod \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " Oct 30 14:50:01.311065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.310688 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8724ac-5136-4a32-b9ab-7f0aa1277938-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8724ac-5136-4a32-b9ab-7f0aa1277938-openshift-service-ca\") pod \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " Oct 30 14:50:01.311065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.310739 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8724ac-5136-4a32-b9ab-7f0aa1277938-kube-api-access-czbkg\" (UniqueName: \"kubernetes.io/projected/1d8724ac-5136-4a32-b9ab-7f0aa1277938-kube-api-access-czbkg\") pod \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\" (UID: \"1d8724ac-5136-4a32-b9ab-7f0aa1277938\") " Oct 30 14:50:01.311424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.311068 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8724ac-5136-4a32-b9ab-7f0aa1277938-openshift-service-ca" pod "1d8724ac-5136-4a32-b9ab-7f0aa1277938" (UID: "1d8724ac-5136-4a32-b9ab-7f0aa1277938"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:01.312655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.312633 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-maas-api-tls" pod "1d8724ac-5136-4a32-b9ab-7f0aa1277938" (UID: "1d8724ac-5136-4a32-b9ab-7f0aa1277938"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:01.312770 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.312748 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-metrics-certs" pod "1d8724ac-5136-4a32-b9ab-7f0aa1277938" (UID: "1d8724ac-5136-4a32-b9ab-7f0aa1277938"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:01.313522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.313499 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8724ac-5136-4a32-b9ab-7f0aa1277938-kube-api-access-czbkg" pod "1d8724ac-5136-4a32-b9ab-7f0aa1277938" (UID: "1d8724ac-5136-4a32-b9ab-7f0aa1277938"). InnerVolumeSpecName "kube-api-access-czbkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:01.352787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.352752 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h"] Oct 30 14:50:01.359495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.359467 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2df0ab/distinct-w0-2df0ab-kserve-59ff959b9d-7sm2h"] Oct 30 14:50:01.411989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.411964 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8724ac-5136-4a32-b9ab-7f0aa1277938-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.411989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.411987 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czbkg\" (UniqueName: \"kubernetes.io/projected/1d8724ac-5136-4a32-b9ab-7f0aa1277938-kube-api-access-czbkg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.412341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.412004 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.412341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.412016 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8724ac-5136-4a32-b9ab-7f0aa1277938-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.429264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.429233 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:50:01.513158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.513086 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-openshift-ca\") pod \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " Oct 30 14:50:01.513158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.513151 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-xks-ca\") pod \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " Oct 30 14:50:01.513415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.513285 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-kube-api-access-cbjmv\" (UniqueName: \"kubernetes.io/projected/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-kube-api-access-cbjmv\") pod \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\" (UID: \"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81\") " Oct 30 14:50:01.513529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.513503 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-openshift-ca" pod "2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81" (UID: "2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:01.513741 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.513725 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.515251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.515224 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-kube-api-access-cbjmv" pod "2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81" (UID: "2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81"). InnerVolumeSpecName "kube-api-access-cbjmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:01.515346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.515290 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-xks-ca" pod "2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81" (UID: "2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:01.610680 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.610642 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx"] Oct 30 14:50:01.612882 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.612851 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0efe43-d56c59457-4rtdx"] Oct 30 14:50:01.614732 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.614710 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:01.614843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:01.614734 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjmv\" (UniqueName: \"kubernetes.io/projected/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81-kube-api-access-cbjmv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:02.102399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.102356 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" podUID="ae95a746-48ba-4686-8f37-ae30206dc275" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:02.273746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.273711 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" Oct 30 14:50:02.273918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.273704 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr" event={"ID":"2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81","Type":"ContainerDied","Data":"4017dcf4181f6ae6b527900996638a71a521042d38b9875a727e2a398c4b7287"} Oct 30 14:50:02.273918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.273888 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4017dcf4181f6ae6b527900996638a71a521042d38b9875a727e2a398c4b7287" Oct 30 14:50:02.315881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.315854 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8724ac-5136-4a32-b9ab-7f0aa1277938" path="/var/lib/kubelet/pods/1d8724ac-5136-4a32-b9ab-7f0aa1277938/volumes" Oct 30 14:50:02.316280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.316266 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a94829-f193-41f6-9cc2-08449ca96d89" path="/var/lib/kubelet/pods/31a94829-f193-41f6-9cc2-08449ca96d89/volumes" Oct 30 14:50:02.316631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.316618 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" path="/var/lib/kubelet/pods/489d0e4e-50f7-48c7-8df9-7482a3455878/volumes" Oct 30 14:50:02.359367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.359312 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt"] Oct 30 14:50:02.367443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.367417 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96"] Oct 30 14:50:02.378073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.378047 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr"] Oct 30 14:50:02.390092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.390071 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx"] Oct 30 14:50:02.987818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:02.987770 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" podUID="5fe90a87-f432-4e8e-8564-45e956c10af7" containerName="payload-processing" containerID="cri-o://d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84" gracePeriod=25 Oct 30 14:50:03.026276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:03.026245 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" podUID="c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" containerName="payload-pre-processing" containerID="cri-o://a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb" gracePeriod=25 Oct 30 14:50:03.126495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:03.126468 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" podUID="7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:03.640007 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:03.639968 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:50:07.232162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:07.232124 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" podUID="650319fd-aeda-4ba4-9594-51ef46a181a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:07.640805 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:07.640730 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" podUID="5fe90a87-f432-4e8e-8564-45e956c10af7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:07.640944 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:07.640730 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" podUID="c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:08.048755 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:08.048722 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn"] Oct 30 14:50:09.161956 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.161918 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh"] Oct 30 14:50:09.162580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162561 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7e1277a3-8129-4d26-9e3e-612d4f7cb14a" containerName="payload-pre-processing" Oct 30 14:50:09.162652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162581 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1277a3-8129-4d26-9e3e-612d4f7cb14a" containerName="payload-pre-processing" Oct 30 14:50:09.162652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162591 5824 cpu_manager.go:414] "containerMap: removing container" podUID="2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81" containerName="revoke-keys" Oct 30 14:50:09.162652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162598 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81" containerName="revoke-keys" Oct 30 14:50:09.162652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162618 5824 cpu_manager.go:414] "containerMap: removing container" podUID="99d95ba4-4fb9-46d2-8b43-08f95c4241bb" containerName="payload-processing" Oct 30 14:50:09.162652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162624 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d95ba4-4fb9-46d2-8b43-08f95c4241bb" containerName="payload-processing" Oct 30 14:50:09.162652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162637 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1d8724ac-5136-4a32-b9ab-7f0aa1277938" containerName="maas-api" Oct 30 14:50:09.162652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162641 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8724ac-5136-4a32-b9ab-7f0aa1277938" containerName="maas-api" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162656 5824 cpu_manager.go:414] "containerMap: removing container" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162661 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162671 5824 cpu_manager.go:414] "containerMap: removing container" podUID="8a41048c-1202-4c97-b5a4-8a3f2c1e76bb" containerName="revoke-keys" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162676 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a41048c-1202-4c97-b5a4-8a3f2c1e76bb" containerName="revoke-keys" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162736 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d95ba4-4fb9-46d2-8b43-08f95c4241bb" containerName="payload-processing" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162745 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8724ac-5136-4a32-b9ab-7f0aa1277938" containerName="maas-api" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162754 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="489d0e4e-50f7-48c7-8df9-7482a3455878" containerName="main" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162761 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a41048c-1202-4c97-b5a4-8a3f2c1e76bb" containerName="revoke-keys" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162770 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1277a3-8129-4d26-9e3e-612d4f7cb14a" containerName="payload-pre-processing" Oct 30 14:50:09.162973 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.162775 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81" containerName="revoke-keys" Oct 30 14:50:09.165937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.165921 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.185254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.185229 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh"] Oct 30 14:50:09.186251 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.186233 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:50:09.194656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.194638 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:50:09.195002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.194983 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:50:09.195002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.194983 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:50:09.290146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.290110 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.290312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.290155 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hz5g\" (UniqueName: \"kubernetes.io/projected/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-kube-api-access-6hz5g\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.290388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.290364 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.391876 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.391849 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.392037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.391912 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.392037 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.391932 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hz5g\" (UniqueName: \"kubernetes.io/projected/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-kube-api-access-6hz5g\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.392619 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.392590 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.394136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.394112 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.418289 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.418228 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hz5g\" (UniqueName: \"kubernetes.io/projected/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-kube-api-access-6hz5g\") pod \"maas-api-revoke-sub-4e287c30-rbcmh\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.479326 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.479301 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:09.779880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:09.779850 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh"] Oct 30 14:50:09.780041 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:50:09.780017 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc67a5f_2b3b_47a2_8729_0ef3fafd5259.slice/crio-6ee235e976218f8362ada0dcd04aa451b150ceb1dc6631bcbee4103511db44cb WatchSource:0}: Error finding container 6ee235e976218f8362ada0dcd04aa451b150ceb1dc6631bcbee4103511db44cb: Status 404 returned error can't find the container with id 6ee235e976218f8362ada0dcd04aa451b150ceb1dc6631bcbee4103511db44cb Oct 30 14:50:10.328366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:10.328241 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" event={"ID":"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259","Type":"ContainerStarted","Data":"5030c2315aa0003c96d905690942c6deca34b888c36b11bba91570a78d5545e4"} Oct 30 14:50:10.328366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:10.328328 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" event={"ID":"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259","Type":"ContainerStarted","Data":"6ee235e976218f8362ada0dcd04aa451b150ceb1dc6631bcbee4103511db44cb"} Oct 30 14:50:10.374951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:10.374900 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" podStartSLOduration=1.374887443 podStartE2EDuration="1.374887443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:50:10.372589607 +0000 UTC m=+3640.173095050" watchObservedRunningTime="2026-10-30 14:50:10.374887443 +0000 UTC m=+3640.175392885" Oct 30 14:50:10.883250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:10.883209 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n"] Oct 30 14:50:10.886922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:10.886899 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:10.905676 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:10.905649 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n"] Oct 30 14:50:11.009902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.009830 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqx2\" (UniqueName: \"kubernetes.io/projected/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-kube-api-access-9tqx2\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.009902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.009863 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.009902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.009882 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.110494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.110465 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqx2\" (UniqueName: \"kubernetes.io/projected/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-kube-api-access-9tqx2\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.110622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.110498 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.110622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.110519 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.111065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.111045 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.112738 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.112718 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.136584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.136563 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqx2\" (UniqueName: \"kubernetes.io/projected/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-kube-api-access-9tqx2\") pod \"maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.199403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.199379 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:11.335858 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.335827 5824 generic.go:356] "Generic (PLEG): container finished" podID="bcc67a5f-2b3b-47a2-8729-0ef3fafd5259" containerID="5030c2315aa0003c96d905690942c6deca34b888c36b11bba91570a78d5545e4" exitCode=0 Oct 30 14:50:11.336280 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.335905 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" event={"ID":"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259","Type":"ContainerDied","Data":"5030c2315aa0003c96d905690942c6deca34b888c36b11bba91570a78d5545e4"} Oct 30 14:50:11.503355 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:11.503326 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n"] Oct 30 14:50:11.503476 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:50:11.503384 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de6e34c_e6fd_4ffd_8a8e_7110fc7d44aa.slice/crio-eb0058cb5d6063aa966c6e85bcd2089fd2acd91dd6fec7cd8f0225758962334a WatchSource:0}: Error finding container eb0058cb5d6063aa966c6e85bcd2089fd2acd91dd6fec7cd8f0225758962334a: Status 404 returned error can't find the container with id eb0058cb5d6063aa966c6e85bcd2089fd2acd91dd6fec7cd8f0225758962334a Oct 30 14:50:12.102020 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.101942 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" podUID="ae95a746-48ba-4686-8f37-ae30206dc275" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:12.343099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.343071 5824 generic.go:356] "Generic (PLEG): container finished" podID="9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa" containerID="acc1d739e0e818bfd8f8752e3221eab219dbe9668d50131b8987e81d1d5d81ac" exitCode=0 Oct 30 14:50:12.343449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.343150 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" event={"ID":"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa","Type":"ContainerDied","Data":"acc1d739e0e818bfd8f8752e3221eab219dbe9668d50131b8987e81d1d5d81ac"} Oct 30 14:50:12.343449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.343229 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" event={"ID":"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa","Type":"ContainerStarted","Data":"eb0058cb5d6063aa966c6e85bcd2089fd2acd91dd6fec7cd8f0225758962334a"} Oct 30 14:50:12.473857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.473834 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:12.630987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.630959 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-xks-ca\") pod \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " Oct 30 14:50:12.631183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.631024 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-kube-api-access-6hz5g\" (UniqueName: \"kubernetes.io/projected/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-kube-api-access-6hz5g\") pod \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " Oct 30 14:50:12.631183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.631106 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-openshift-ca\") pod \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\" (UID: \"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259\") " Oct 30 14:50:12.631600 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.631571 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-openshift-ca" pod "bcc67a5f-2b3b-47a2-8729-0ef3fafd5259" (UID: "bcc67a5f-2b3b-47a2-8729-0ef3fafd5259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:12.633413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.633385 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-xks-ca" pod "bcc67a5f-2b3b-47a2-8729-0ef3fafd5259" (UID: "bcc67a5f-2b3b-47a2-8729-0ef3fafd5259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:12.633523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.633408 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-kube-api-access-6hz5g" pod "bcc67a5f-2b3b-47a2-8729-0ef3fafd5259" (UID: "bcc67a5f-2b3b-47a2-8729-0ef3fafd5259"). InnerVolumeSpecName "kube-api-access-6hz5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:12.732436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.732408 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hz5g\" (UniqueName: \"kubernetes.io/projected/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-kube-api-access-6hz5g\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:12.732436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.732433 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:12.732582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:12.732443 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:13.126405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.126326 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" podUID="7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:13.313912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.313874 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" podUID="81282091-cc28-4f1d-b6bd-aa68cadd63c0" containerName="payload-pre-processing" containerID="cri-o://b43edd5ecf080e5a0b9b98f478e9592175e0ea2dba14966a05ba3e0204af12da" gracePeriod=25 Oct 30 14:50:13.350621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.350594 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" Oct 30 14:50:13.350989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.350615 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh" event={"ID":"bcc67a5f-2b3b-47a2-8729-0ef3fafd5259","Type":"ContainerDied","Data":"6ee235e976218f8362ada0dcd04aa451b150ceb1dc6631bcbee4103511db44cb"} Oct 30 14:50:13.350989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.350663 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ee235e976218f8362ada0dcd04aa451b150ceb1dc6631bcbee4103511db44cb" Oct 30 14:50:13.493576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.493551 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:13.641532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.641492 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-xks-ca\") pod \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " Oct 30 14:50:13.641708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.641600 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-openshift-ca\") pod \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " Oct 30 14:50:13.641757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.641732 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-kube-api-access-9tqx2\" (UniqueName: \"kubernetes.io/projected/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-kube-api-access-9tqx2\") pod \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\" (UID: \"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa\") " Oct 30 14:50:13.642076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.642041 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-openshift-ca" pod "9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa" (UID: "9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:13.642198 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.642165 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:13.643555 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.643533 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-xks-ca" pod "9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa" (UID: "9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:13.643699 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.643680 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-kube-api-access-9tqx2" pod "9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa" (UID: "9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa"). InnerVolumeSpecName "kube-api-access-9tqx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:13.743655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.743629 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tqx2\" (UniqueName: \"kubernetes.io/projected/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-kube-api-access-9tqx2\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:13.743655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:13.743657 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:14.035075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:14.034998 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5"] Oct 30 14:50:14.041295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:14.041263 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf056ee6-q65v5"] Oct 30 14:50:14.316761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:14.316686 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51621a4-9cc5-4fb6-be52-903bee548a74" path="/var/lib/kubelet/pods/c51621a4-9cc5-4fb6-be52-903bee548a74/volumes" Oct 30 14:50:14.358750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:14.358724 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" Oct 30 14:50:14.358750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:14.358737 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n" event={"ID":"9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa","Type":"ContainerDied","Data":"eb0058cb5d6063aa966c6e85bcd2089fd2acd91dd6fec7cd8f0225758962334a"} Oct 30 14:50:14.359182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:14.358793 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb0058cb5d6063aa966c6e85bcd2089fd2acd91dd6fec7cd8f0225758962334a" Oct 30 14:50:14.364073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:14.364030 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4"] Oct 30 14:50:14.366361 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:14.366338 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2df0ab-5lwm4"] Oct 30 14:50:15.688982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:15.688948 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh"] Oct 30 14:50:16.316665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.316633 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d94f8ca-20da-4151-ab97-4ad52c787fb1" path="/var/lib/kubelet/pods/2d94f8ca-20da-4151-ab97-4ad52c787fb1/volumes" Oct 30 14:50:16.376001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.375966 5824 generic.go:356] "Generic (PLEG): container finished" podID="650319fd-aeda-4ba4-9594-51ef46a181a9" containerID="a87132cdbc0e64649aebf220ba321cfdb637e8976020fb3780651545b426ff68" exitCode=0 Oct 30 14:50:16.376171 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.376041 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" event={"ID":"650319fd-aeda-4ba4-9594-51ef46a181a9","Type":"ContainerDied","Data":"a87132cdbc0e64649aebf220ba321cfdb637e8976020fb3780651545b426ff68"} Oct 30 14:50:16.494297 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.494227 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:50:16.672797 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.672769 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/650319fd-aeda-4ba4-9594-51ef46a181a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650319fd-aeda-4ba4-9594-51ef46a181a9-plugins-config-volume\") pod \"650319fd-aeda-4ba4-9594-51ef46a181a9\" (UID: \"650319fd-aeda-4ba4-9594-51ef46a181a9\") " Oct 30 14:50:16.672982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.672907 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650319fd-aeda-4ba4-9594-51ef46a181a9-kube-api-access-rhhqj\" (UniqueName: \"kubernetes.io/projected/650319fd-aeda-4ba4-9594-51ef46a181a9-kube-api-access-rhhqj\") pod \"650319fd-aeda-4ba4-9594-51ef46a181a9\" (UID: \"650319fd-aeda-4ba4-9594-51ef46a181a9\") " Oct 30 14:50:16.673685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.673441 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650319fd-aeda-4ba4-9594-51ef46a181a9-plugins-config-volume" pod "650319fd-aeda-4ba4-9594-51ef46a181a9" (UID: "650319fd-aeda-4ba4-9594-51ef46a181a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:16.685212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.676247 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650319fd-aeda-4ba4-9594-51ef46a181a9-kube-api-access-rhhqj" pod "650319fd-aeda-4ba4-9594-51ef46a181a9" (UID: "650319fd-aeda-4ba4-9594-51ef46a181a9"). InnerVolumeSpecName "kube-api-access-rhhqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:16.774421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.774364 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/650319fd-aeda-4ba4-9594-51ef46a181a9-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:16.774421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:16.774388 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhqj\" (UniqueName: \"kubernetes.io/projected/650319fd-aeda-4ba4-9594-51ef46a181a9-kube-api-access-rhhqj\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.381546 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.381509 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" containerID="cri-o://1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777" gracePeriod=45 Oct 30 14:50:17.387490 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.387457 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" containerID="cri-o://2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec" gracePeriod=45 Oct 30 14:50:17.389312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.389288 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" Oct 30 14:50:17.389441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.389310 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49" event={"ID":"650319fd-aeda-4ba4-9594-51ef46a181a9","Type":"ContainerDied","Data":"ca8f346ff9c08b5ebce5d7180a7c221bec4f7fd8896edcd35cc38299a3128cc7"} Oct 30 14:50:17.389441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.389379 5824 scope.go:122] "RemoveContainer" containerID="a87132cdbc0e64649aebf220ba321cfdb637e8976020fb3780651545b426ff68" Oct 30 14:50:17.397150 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.397124 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" containerID="cri-o://0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b" gracePeriod=45 Oct 30 14:50:17.408513 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.408490 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" containerID="cri-o://69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630" gracePeriod=45 Oct 30 14:50:17.475606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.475580 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Oct 30 14:50:17.655060 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.654079 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" podUID="5fe90a87-f432-4e8e-8564-45e956c10af7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:17.656155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.655913 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" podUID="c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:17.697378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.697333 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49"] Oct 30 14:50:17.704321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.703990 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2df0ab-5649f578f6-lbf49"] Oct 30 14:50:17.800094 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.799832 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:50:17.804656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.804575 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:50:17.807912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.807854 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:50:17.815697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.815673 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:50:17.857993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.857891 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:50:17.860652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.860623 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:50:17.891628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891543 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-home\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-home\") pod \"6eddaacf-b1dc-41be-96fb-4ca539929156\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " Oct 30 14:50:17.891628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891614 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-plugins-config-volume\") pod \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\" (UID: \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\") " Oct 30 14:50:17.891850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891654 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-tmp-dir\") pod \"057f7e4f-090a-42ab-92a1-cc9359882375\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " Oct 30 14:50:17.891850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891684 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae95a746-48ba-4686-8f37-ae30206dc275-kube-api-access-wjcjr\" (UniqueName: \"kubernetes.io/projected/ae95a746-48ba-4686-8f37-ae30206dc275-kube-api-access-wjcjr\") pod \"ae95a746-48ba-4686-8f37-ae30206dc275\" (UID: \"ae95a746-48ba-4686-8f37-ae30206dc275\") " Oct 30 14:50:17.891850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891720 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-dshm\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-dshm\") pod \"349413fe-1608-4cd2-90d0-3dca7efc3606\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " Oct 30 14:50:17.891850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891746 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-kube-api-access-bp9dk\" (UniqueName: \"kubernetes.io/projected/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-kube-api-access-bp9dk\") pod \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\" (UID: \"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf\") " Oct 30 14:50:17.891850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891778 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-model-cache\") pod \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " Oct 30 14:50:17.891850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891796 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-model-cache\") pod \"057f7e4f-090a-42ab-92a1-cc9359882375\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " Oct 30 14:50:17.891850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891816 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057f7e4f-090a-42ab-92a1-cc9359882375-tls-certs\" (UniqueName: \"kubernetes.io/secret/057f7e4f-090a-42ab-92a1-cc9359882375-tls-certs\") pod \"057f7e4f-090a-42ab-92a1-cc9359882375\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " Oct 30 14:50:17.891850 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891833 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eddaacf-b1dc-41be-96fb-4ca539929156-kube-api-access-8q9sv\" (UniqueName: \"kubernetes.io/projected/6eddaacf-b1dc-41be-96fb-4ca539929156-kube-api-access-8q9sv\") pod \"6eddaacf-b1dc-41be-96fb-4ca539929156\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891859 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-dshm\") pod \"6eddaacf-b1dc-41be-96fb-4ca539929156\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891882 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-model-cache\") pod \"349413fe-1608-4cd2-90d0-3dca7efc3606\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891908 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-model-cache\") pod \"6eddaacf-b1dc-41be-96fb-4ca539929156\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891952 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057f7e4f-090a-42ab-92a1-cc9359882375-kube-api-access-qsgkk\" (UniqueName: \"kubernetes.io/projected/057f7e4f-090a-42ab-92a1-cc9359882375-kube-api-access-qsgkk\") pod \"057f7e4f-090a-42ab-92a1-cc9359882375\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.891970 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349413fe-1608-4cd2-90d0-3dca7efc3606-tls-certs\" (UniqueName: \"kubernetes.io/secret/349413fe-1608-4cd2-90d0-3dca7efc3606-tls-certs\") pod \"349413fe-1608-4cd2-90d0-3dca7efc3606\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892077 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tls-certs\") pod \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892117 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eddaacf-b1dc-41be-96fb-4ca539929156-tls-certs\" (UniqueName: \"kubernetes.io/secret/6eddaacf-b1dc-41be-96fb-4ca539929156-tls-certs\") pod \"6eddaacf-b1dc-41be-96fb-4ca539929156\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892151 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-home\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-home\") pod \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892180 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-dshm\") pod \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892229 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-home\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-home\") pod \"349413fe-1608-4cd2-90d0-3dca7efc3606\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892311 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-tmp-dir\") pod \"349413fe-1608-4cd2-90d0-3dca7efc3606\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " Oct 30 14:50:17.892362 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892341 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349413fe-1608-4cd2-90d0-3dca7efc3606-kube-api-access-kjsk8\" (UniqueName: \"kubernetes.io/projected/349413fe-1608-4cd2-90d0-3dca7efc3606-kube-api-access-kjsk8\") pod \"349413fe-1608-4cd2-90d0-3dca7efc3606\" (UID: \"349413fe-1608-4cd2-90d0-3dca7efc3606\") " Oct 30 14:50:17.892942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892373 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-dshm\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-dshm\") pod \"057f7e4f-090a-42ab-92a1-cc9359882375\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " Oct 30 14:50:17.892942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892453 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-home\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-home\") pod \"057f7e4f-090a-42ab-92a1-cc9359882375\" (UID: \"057f7e4f-090a-42ab-92a1-cc9359882375\") " Oct 30 14:50:17.892942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892496 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-tmp-dir\") pod \"6eddaacf-b1dc-41be-96fb-4ca539929156\" (UID: \"6eddaacf-b1dc-41be-96fb-4ca539929156\") " Oct 30 14:50:17.892942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892526 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4459dc7d-8724-4e45-a3fd-d95cc04365fd-kube-api-access-f2wk2\" (UniqueName: \"kubernetes.io/projected/4459dc7d-8724-4e45-a3fd-d95cc04365fd-kube-api-access-f2wk2\") pod \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " Oct 30 14:50:17.892942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892555 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tmp-dir\") pod \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\" (UID: \"4459dc7d-8724-4e45-a3fd-d95cc04365fd\") " Oct 30 14:50:17.892942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.892621 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae95a746-48ba-4686-8f37-ae30206dc275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae95a746-48ba-4686-8f37-ae30206dc275-plugins-config-volume\") pod \"ae95a746-48ba-4686-8f37-ae30206dc275\" (UID: \"ae95a746-48ba-4686-8f37-ae30206dc275\") " Oct 30 14:50:17.893500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.893473 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae95a746-48ba-4686-8f37-ae30206dc275-plugins-config-volume" pod "ae95a746-48ba-4686-8f37-ae30206dc275" (UID: "ae95a746-48ba-4686-8f37-ae30206dc275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:17.894003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.893974 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-home" pod "6eddaacf-b1dc-41be-96fb-4ca539929156" (UID: "6eddaacf-b1dc-41be-96fb-4ca539929156"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.894523 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.894361 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-plugins-config-volume" pod "7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" (UID: "7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:17.895057 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.894502 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-model-cache" pod "057f7e4f-090a-42ab-92a1-cc9359882375" (UID: "057f7e4f-090a-42ab-92a1-cc9359882375"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.895208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.894664 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-tmp-dir" pod "057f7e4f-090a-42ab-92a1-cc9359882375" (UID: "057f7e4f-090a-42ab-92a1-cc9359882375"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.895552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.895529 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-home" pod "4459dc7d-8724-4e45-a3fd-d95cc04365fd" (UID: "4459dc7d-8724-4e45-a3fd-d95cc04365fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.896043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.895818 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-tmp-dir" pod "349413fe-1608-4cd2-90d0-3dca7efc3606" (UID: "349413fe-1608-4cd2-90d0-3dca7efc3606"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.896393 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.896372 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-model-cache" pod "4459dc7d-8724-4e45-a3fd-d95cc04365fd" (UID: "4459dc7d-8724-4e45-a3fd-d95cc04365fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.898073 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.898043 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349413fe-1608-4cd2-90d0-3dca7efc3606-kube-api-access-kjsk8" pod "349413fe-1608-4cd2-90d0-3dca7efc3606" (UID: "349413fe-1608-4cd2-90d0-3dca7efc3606"). InnerVolumeSpecName "kube-api-access-kjsk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:17.898569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.898525 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tls-certs" pod "4459dc7d-8724-4e45-a3fd-d95cc04365fd" (UID: "4459dc7d-8724-4e45-a3fd-d95cc04365fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:17.898828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.898802 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-model-cache" pod "6eddaacf-b1dc-41be-96fb-4ca539929156" (UID: "6eddaacf-b1dc-41be-96fb-4ca539929156"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.899023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.898997 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-model-cache" pod "349413fe-1608-4cd2-90d0-3dca7efc3606" (UID: "349413fe-1608-4cd2-90d0-3dca7efc3606"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.899381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.899205 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-tmp-dir" pod "6eddaacf-b1dc-41be-96fb-4ca539929156" (UID: "6eddaacf-b1dc-41be-96fb-4ca539929156"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.899588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.899561 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-home" pod "057f7e4f-090a-42ab-92a1-cc9359882375" (UID: "057f7e4f-090a-42ab-92a1-cc9359882375"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.900176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.900157 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-home" pod "349413fe-1608-4cd2-90d0-3dca7efc3606" (UID: "349413fe-1608-4cd2-90d0-3dca7efc3606"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.902018 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.901998 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-dshm" pod "349413fe-1608-4cd2-90d0-3dca7efc3606" (UID: "349413fe-1608-4cd2-90d0-3dca7efc3606"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.902110 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.902070 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-kube-api-access-bp9dk" pod "7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" (UID: "7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf"). InnerVolumeSpecName "kube-api-access-bp9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:17.902172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.902124 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-dshm" pod "4459dc7d-8724-4e45-a3fd-d95cc04365fd" (UID: "4459dc7d-8724-4e45-a3fd-d95cc04365fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.902172 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.902144 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349413fe-1608-4cd2-90d0-3dca7efc3606-tls-certs" pod "349413fe-1608-4cd2-90d0-3dca7efc3606" (UID: "349413fe-1608-4cd2-90d0-3dca7efc3606"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:17.902310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.902202 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae95a746-48ba-4686-8f37-ae30206dc275-kube-api-access-wjcjr" pod "ae95a746-48ba-4686-8f37-ae30206dc275" (UID: "ae95a746-48ba-4686-8f37-ae30206dc275"). InnerVolumeSpecName "kube-api-access-wjcjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:17.902310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.902176 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-dshm" pod "6eddaacf-b1dc-41be-96fb-4ca539929156" (UID: "6eddaacf-b1dc-41be-96fb-4ca539929156"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.902468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.902441 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tmp-dir" pod "4459dc7d-8724-4e45-a3fd-d95cc04365fd" (UID: "4459dc7d-8724-4e45-a3fd-d95cc04365fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.902899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.902874 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eddaacf-b1dc-41be-96fb-4ca539929156-kube-api-access-8q9sv" pod "6eddaacf-b1dc-41be-96fb-4ca539929156" (UID: "6eddaacf-b1dc-41be-96fb-4ca539929156"). InnerVolumeSpecName "kube-api-access-8q9sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:17.903128 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.903103 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057f7e4f-090a-42ab-92a1-cc9359882375-tls-certs" pod "057f7e4f-090a-42ab-92a1-cc9359882375" (UID: "057f7e4f-090a-42ab-92a1-cc9359882375"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:17.903329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.903181 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eddaacf-b1dc-41be-96fb-4ca539929156-tls-certs" pod "6eddaacf-b1dc-41be-96fb-4ca539929156" (UID: "6eddaacf-b1dc-41be-96fb-4ca539929156"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:17.903329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.903281 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057f7e4f-090a-42ab-92a1-cc9359882375-kube-api-access-qsgkk" pod "057f7e4f-090a-42ab-92a1-cc9359882375" (UID: "057f7e4f-090a-42ab-92a1-cc9359882375"). InnerVolumeSpecName "kube-api-access-qsgkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:17.903645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.903625 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-dshm" pod "057f7e4f-090a-42ab-92a1-cc9359882375" (UID: "057f7e4f-090a-42ab-92a1-cc9359882375"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:17.904807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.904786 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4459dc7d-8724-4e45-a3fd-d95cc04365fd-kube-api-access-f2wk2" pod "4459dc7d-8724-4e45-a3fd-d95cc04365fd" (UID: "4459dc7d-8724-4e45-a3fd-d95cc04365fd"). InnerVolumeSpecName "kube-api-access-f2wk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:17.994969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.994933 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.994969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.994972 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.994988 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2wk2\" (UniqueName: \"kubernetes.io/projected/4459dc7d-8724-4e45-a3fd-d95cc04365fd-kube-api-access-f2wk2\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995002 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995016 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae95a746-48ba-4686-8f37-ae30206dc275-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995028 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995041 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995054 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995068 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjcjr\" (UniqueName: \"kubernetes.io/projected/ae95a746-48ba-4686-8f37-ae30206dc275-kube-api-access-wjcjr\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995082 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995095 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp9dk\" (UniqueName: \"kubernetes.io/projected/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf-kube-api-access-bp9dk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995109 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995129 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995141 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057f7e4f-090a-42ab-92a1-cc9359882375-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995153 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q9sv\" (UniqueName: \"kubernetes.io/projected/6eddaacf-b1dc-41be-96fb-4ca539929156-kube-api-access-8q9sv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995167 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995180 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995210 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eddaacf-b1dc-41be-96fb-4ca539929156-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995217 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995223 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsgkk\" (UniqueName: \"kubernetes.io/projected/057f7e4f-090a-42ab-92a1-cc9359882375-kube-api-access-qsgkk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995237 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349413fe-1608-4cd2-90d0-3dca7efc3606-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995252 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4459dc7d-8724-4e45-a3fd-d95cc04365fd-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995265 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eddaacf-b1dc-41be-96fb-4ca539929156-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995279 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995290 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4459dc7d-8724-4e45-a3fd-d95cc04365fd-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995303 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995315 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349413fe-1608-4cd2-90d0-3dca7efc3606-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995330 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjsk8\" (UniqueName: \"kubernetes.io/projected/349413fe-1608-4cd2-90d0-3dca7efc3606-kube-api-access-kjsk8\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:17.995912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:17.995345 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057f7e4f-090a-42ab-92a1-cc9359882375-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:18.315737 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.315701 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650319fd-aeda-4ba4-9594-51ef46a181a9" path="/var/lib/kubelet/pods/650319fd-aeda-4ba4-9594-51ef46a181a9/volumes" Oct 30 14:50:18.396578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.396544 5824 generic.go:356] "Generic (PLEG): container finished" podID="057f7e4f-090a-42ab-92a1-cc9359882375" containerID="69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630" exitCode=0 Oct 30 14:50:18.396689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.396582 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" Oct 30 14:50:18.396689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.396622 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" event={"ID":"057f7e4f-090a-42ab-92a1-cc9359882375","Type":"ContainerDied","Data":"69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630"} Oct 30 14:50:18.396689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.396665 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx" event={"ID":"057f7e4f-090a-42ab-92a1-cc9359882375","Type":"ContainerDied","Data":"95988073dd6feab4f0d8f76033a7a3f63c1cf4849a8edac50ba4bc410480597f"} Oct 30 14:50:18.396689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.396684 5824 scope.go:122] "RemoveContainer" containerID="69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630" Oct 30 14:50:18.398083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.398062 5824 generic.go:356] "Generic (PLEG): container finished" podID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerID="0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b" exitCode=0 Oct 30 14:50:18.398146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.398105 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" Oct 30 14:50:18.398209 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.398102 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" event={"ID":"6eddaacf-b1dc-41be-96fb-4ca539929156","Type":"ContainerDied","Data":"0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b"} Oct 30 14:50:18.398270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.398225 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr" event={"ID":"6eddaacf-b1dc-41be-96fb-4ca539929156","Type":"ContainerDied","Data":"46a83b5e1455fad6255b5ee3088802cb49d8a7b30242befd0fe92feea55b5fcf"} Oct 30 14:50:18.399772 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.399751 5824 generic.go:356] "Generic (PLEG): container finished" podID="ae95a746-48ba-4686-8f37-ae30206dc275" containerID="0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5" exitCode=0 Oct 30 14:50:18.399883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.399796 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" Oct 30 14:50:18.399883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.399812 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" event={"ID":"ae95a746-48ba-4686-8f37-ae30206dc275","Type":"ContainerDied","Data":"0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5"} Oct 30 14:50:18.399883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.399848 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2" event={"ID":"ae95a746-48ba-4686-8f37-ae30206dc275","Type":"ContainerDied","Data":"c9ef7c74c78429b547673c2bcf8c6f412074f32c070bd804e6858e47f7f905f0"} Oct 30 14:50:18.401402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.401379 5824 generic.go:356] "Generic (PLEG): container finished" podID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerID="1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777" exitCode=0 Oct 30 14:50:18.401493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.401430 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" Oct 30 14:50:18.401493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.401459 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" event={"ID":"4459dc7d-8724-4e45-a3fd-d95cc04365fd","Type":"ContainerDied","Data":"1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777"} Oct 30 14:50:18.401493 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.401489 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" event={"ID":"4459dc7d-8724-4e45-a3fd-d95cc04365fd","Type":"ContainerDied","Data":"3eabe30698e3da8d851539a629b1d1f21109d638d9d68cfca3548a401d4ba303"} Oct 30 14:50:18.402952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.402934 5824 generic.go:356] "Generic (PLEG): container finished" podID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerID="2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec" exitCode=0 Oct 30 14:50:18.403053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.402975 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" Oct 30 14:50:18.403053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.402999 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" event={"ID":"349413fe-1608-4cd2-90d0-3dca7efc3606","Type":"ContainerDied","Data":"2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec"} Oct 30 14:50:18.403053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.403024 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96" event={"ID":"349413fe-1608-4cd2-90d0-3dca7efc3606","Type":"ContainerDied","Data":"08eefea885521322752d40e347f760a254811186cca6fd400ca24670d378ca34"} Oct 30 14:50:18.404319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.404301 5824 generic.go:356] "Generic (PLEG): container finished" podID="7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" containerID="aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b" exitCode=0 Oct 30 14:50:18.404389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.404381 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" event={"ID":"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf","Type":"ContainerDied","Data":"aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b"} Oct 30 14:50:18.404435 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.404408 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" event={"ID":"7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf","Type":"ContainerDied","Data":"6cb89222fb3ef242a327f33933fee444f912bbc138f355f83ca6e113339ec6e6"} Oct 30 14:50:18.404468 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.404453 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66" Oct 30 14:50:18.408323 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.408304 5824 scope.go:122] "RemoveContainer" containerID="69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630" Oct 30 14:50:18.408613 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:18.408581 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630\": container with ID starting with 69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630 not found: ID does not exist" containerID="69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630" Oct 30 14:50:18.408693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.408612 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630"} err="failed to get container status \"69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630\": rpc error: code = NotFound desc = could not find container \"69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630\": container with ID starting with 69671f014d7435003e2ba1594a7ee7d25055b874898746c2bde35274a3bfe630 not found: ID does not exist" Oct 30 14:50:18.408693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.408629 5824 scope.go:122] "RemoveContainer" containerID="0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b" Oct 30 14:50:18.416867 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.416851 5824 scope.go:122] "RemoveContainer" containerID="0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b" Oct 30 14:50:18.417109 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:18.417091 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b\": container with ID starting with 0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b not found: ID does not exist" containerID="0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b" Oct 30 14:50:18.417183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.417114 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b"} err="failed to get container status \"0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b\": rpc error: code = NotFound desc = could not find container \"0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b\": container with ID starting with 0b4c4ab248e25bef0f39b37804c01eb6638651427332d91c196b6baab642565b not found: ID does not exist" Oct 30 14:50:18.417183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.417131 5824 scope.go:122] "RemoveContainer" containerID="0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5" Oct 30 14:50:18.425236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.425216 5824 scope.go:122] "RemoveContainer" containerID="0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5" Oct 30 14:50:18.425504 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:18.425486 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5\": container with ID starting with 0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5 not found: ID does not exist" containerID="0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5" Oct 30 14:50:18.425560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.425508 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5"} err="failed to get container status \"0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5\": rpc error: code = NotFound desc = could not find container \"0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5\": container with ID starting with 0c4ee3d2ee287a12e2254de326c185c9d4c0ecb5970c14f0432053352103dff5 not found: ID does not exist" Oct 30 14:50:18.425560 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.425525 5824 scope.go:122] "RemoveContainer" containerID="1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777" Oct 30 14:50:18.433552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.433537 5824 scope.go:122] "RemoveContainer" containerID="1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777" Oct 30 14:50:18.433785 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:18.433771 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777\": container with ID starting with 1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777 not found: ID does not exist" containerID="1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777" Oct 30 14:50:18.433825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.433790 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777"} err="failed to get container status \"1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777\": rpc error: code = NotFound desc = could not find container \"1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777\": container with ID starting with 1aa14cca832e10422fed56f7f1a1cef5c582fb1f9a8a2fe83c1b1fc000c95777 not found: ID does not exist" Oct 30 14:50:18.433825 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.433809 5824 scope.go:122] "RemoveContainer" containerID="2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec" Oct 30 14:50:18.441646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.441626 5824 scope.go:122] "RemoveContainer" containerID="2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec" Oct 30 14:50:18.441923 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:18.441903 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec\": container with ID starting with 2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec not found: ID does not exist" containerID="2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec" Oct 30 14:50:18.441987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.441927 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec"} err="failed to get container status \"2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec\": rpc error: code = NotFound desc = could not find container \"2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec\": container with ID starting with 2d69ed1327c595e4c4485514d36c19728ba7a821f0d946d02d71db0cf25accec not found: ID does not exist" Oct 30 14:50:18.441987 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.441941 5824 scope.go:122] "RemoveContainer" containerID="aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b" Oct 30 14:50:18.449879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.449859 5824 scope.go:122] "RemoveContainer" containerID="aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b" Oct 30 14:50:18.450139 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:18.450121 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b\": container with ID starting with aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b not found: ID does not exist" containerID="aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b" Oct 30 14:50:18.450213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.450145 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b"} err="failed to get container status \"aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b\": rpc error: code = NotFound desc = could not find container \"aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b\": container with ID starting with aec8e1cfdbad2d52d77ffec2f7ded2598f9531287bbe09bd831e8ba6676df53b not found: ID does not exist" Oct 30 14:50:18.455585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.455561 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96"] Oct 30 14:50:18.457740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.457720 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-w4-46b13b-kserve-7f9d98b55f-xfk96"] Oct 30 14:50:18.497489 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.497466 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt"] Oct 30 14:50:18.502577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.502557 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt"] Oct 30 14:50:18.543904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.543878 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2"] Oct 30 14:50:18.552310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.552292 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0efe43-7474755dbc-nqdc2"] Oct 30 14:50:18.591540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.591487 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx"] Oct 30 14:50:18.597350 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.597331 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/simulator-w4-46b13b-kserve-778dd784df-c9mnx"] Oct 30 14:50:18.638139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.638111 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66"] Oct 30 14:50:18.643108 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.643088 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0efe43-54c6bd4ff6-pdj66"] Oct 30 14:50:18.682078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.682058 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr"] Oct 30 14:50:18.687302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.687283 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-46b13b/premium-w4-46b13b-kserve-6c64c684f6-z87tr"] Oct 30 14:50:18.758799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:18.758762 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-46b13b/distinct-2-w4-46b13b-kserve-898c49c88-kq6wt" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 14:50:19.243751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:19.243716 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" podUID="81282091-cc28-4f1d-b6bd-aa68cadd63c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:20.315228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:20.315175 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" path="/var/lib/kubelet/pods/057f7e4f-090a-42ab-92a1-cc9359882375/volumes" Oct 30 14:50:20.315620 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:20.315603 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" path="/var/lib/kubelet/pods/349413fe-1608-4cd2-90d0-3dca7efc3606/volumes" Oct 30 14:50:20.316005 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:20.315991 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" path="/var/lib/kubelet/pods/4459dc7d-8724-4e45-a3fd-d95cc04365fd/volumes" Oct 30 14:50:20.316681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:20.316665 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" path="/var/lib/kubelet/pods/6eddaacf-b1dc-41be-96fb-4ca539929156/volumes" Oct 30 14:50:20.317041 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:20.317027 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" path="/var/lib/kubelet/pods/7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf/volumes" Oct 30 14:50:20.317404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:20.317390 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae95a746-48ba-4686-8f37-ae30206dc275" path="/var/lib/kubelet/pods/ae95a746-48ba-4686-8f37-ae30206dc275/volumes" Oct 30 14:50:22.422662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.422628 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4"] Oct 30 14:50:22.456074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.456042 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z"] Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506165 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm"] Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506784 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506803 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506827 5824 cpu_manager.go:414] "containerMap: removing container" podUID="650319fd-aeda-4ba4-9594-51ef46a181a9" containerName="payload-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506835 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650319fd-aeda-4ba4-9594-51ef46a181a9" containerName="payload-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506848 5824 cpu_manager.go:414] "containerMap: removing container" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506854 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506880 5824 cpu_manager.go:414] "containerMap: removing container" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506886 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506908 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ae95a746-48ba-4686-8f37-ae30206dc275" containerName="payload-pre-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506914 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae95a746-48ba-4686-8f37-ae30206dc275" containerName="payload-pre-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506931 5824 cpu_manager.go:414] "containerMap: removing container" podUID="7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" containerName="payload-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506938 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" containerName="payload-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506954 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506961 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506970 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa" containerName="revoke-keys" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506976 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa" containerName="revoke-keys" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.506994 5824 cpu_manager.go:414] "containerMap: removing container" podUID="bcc67a5f-2b3b-47a2-8729-0ef3fafd5259" containerName="revoke-keys" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507000 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc67a5f-2b3b-47a2-8729-0ef3fafd5259" containerName="revoke-keys" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507083 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="349413fe-1608-4cd2-90d0-3dca7efc3606" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507098 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae95a746-48ba-4686-8f37-ae30206dc275" containerName="payload-pre-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507107 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f31e1a3-877c-476f-bbf8-7eef6fcc3bbf" containerName="payload-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507124 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4459dc7d-8724-4e45-a3fd-d95cc04365fd" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507132 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa" containerName="revoke-keys" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507141 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc67a5f-2b3b-47a2-8729-0ef3fafd5259" containerName="revoke-keys" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507148 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eddaacf-b1dc-41be-96fb-4ca539929156" containerName="main" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507159 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="650319fd-aeda-4ba4-9594-51ef46a181a9" containerName="payload-processing" Oct 30 14:50:22.507223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.507167 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="057f7e4f-090a-42ab-92a1-cc9359882375" containerName="main" Oct 30 14:50:22.514240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.514217 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.532553 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.532524 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm"] Oct 30 14:50:22.532780 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.532764 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:50:22.532846 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.532812 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:50:22.607526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.607494 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:50:22.607718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.607605 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:50:22.608120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.608095 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x"] Oct 30 14:50:22.611913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.611891 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.632223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.632162 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x"] Oct 30 14:50:22.639898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.639727 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.639898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.639815 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wm7l\" (UniqueName: \"kubernetes.io/projected/3584ff8d-b5ca-413c-8abb-52ad766ae183-kube-api-access-6wm7l\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.640121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.639904 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.741408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.741374 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.741624 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.741427 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.741624 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.741477 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.741768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.741623 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wm7l\" (UniqueName: \"kubernetes.io/projected/3584ff8d-b5ca-413c-8abb-52ad766ae183-kube-api-access-6wm7l\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.741768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.741656 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdxt\" (UniqueName: \"kubernetes.io/projected/af496e84-e61e-48cc-9a60-234c05473f92-kube-api-access-7mdxt\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.741768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.741747 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.742285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.742254 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.744315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.744288 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.781482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.781445 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wm7l\" (UniqueName: \"kubernetes.io/projected/3584ff8d-b5ca-413c-8abb-52ad766ae183-kube-api-access-6wm7l\") pod \"maas-api-revoke-sub-c002ceec-g7ckm\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.796134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.795057 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t"] Oct 30 14:50:22.796134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.795345 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" containerID="cri-o://761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3" gracePeriod=30 Oct 30 14:50:22.834697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.834667 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:22.844413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.844373 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdxt\" (UniqueName: \"kubernetes.io/projected/af496e84-e61e-48cc-9a60-234c05473f92-kube-api-access-7mdxt\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.844554 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.844538 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.844631 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.844579 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.846787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.845405 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.849526 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.849499 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.872318 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.872292 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdxt\" (UniqueName: \"kubernetes.io/projected/af496e84-e61e-48cc-9a60-234c05473f92-kube-api-access-7mdxt\") pod \"maas-api-revoke-sub-e848ef7d-68q7x\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:22.928446 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:22.928415 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:23.087812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.087773 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg"] Oct 30 14:50:23.089774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.089750 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-46b13b-xf4fg"] Oct 30 14:50:23.168203 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:50:23.167713 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3584ff8d_b5ca_413c_8abb_52ad766ae183.slice/crio-b5cfb791bbaaec5573688fe998bf2d0cff93b0d8760fabeafc5d7002bc36e149 WatchSource:0}: Error finding container b5cfb791bbaaec5573688fe998bf2d0cff93b0d8760fabeafc5d7002bc36e149: Status 404 returned error can't find the container with id b5cfb791bbaaec5573688fe998bf2d0cff93b0d8760fabeafc5d7002bc36e149 Oct 30 14:50:23.175667 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.169750 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm"] Oct 30 14:50:23.249519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.249424 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x"] Oct 30 14:50:23.356462 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.356439 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:50:23.363239 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.363218 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:50:23.452262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.452212 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-kube-api-access-jr9mg\" (UniqueName: \"kubernetes.io/projected/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-kube-api-access-jr9mg\") pod \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\" (UID: \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\") " Oct 30 14:50:23.452834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.452807 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe90a87-f432-4e8e-8564-45e956c10af7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe90a87-f432-4e8e-8564-45e956c10af7-plugins-config-volume\") pod \"5fe90a87-f432-4e8e-8564-45e956c10af7\" (UID: \"5fe90a87-f432-4e8e-8564-45e956c10af7\") " Oct 30 14:50:23.453009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.452994 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-plugins-config-volume\") pod \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\" (UID: \"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7\") " Oct 30 14:50:23.453232 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.453182 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe90a87-f432-4e8e-8564-45e956c10af7-kube-api-access-6g598\" (UniqueName: \"kubernetes.io/projected/5fe90a87-f432-4e8e-8564-45e956c10af7-kube-api-access-6g598\") pod \"5fe90a87-f432-4e8e-8564-45e956c10af7\" (UID: \"5fe90a87-f432-4e8e-8564-45e956c10af7\") " Oct 30 14:50:23.453427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.453399 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe90a87-f432-4e8e-8564-45e956c10af7-plugins-config-volume" pod "5fe90a87-f432-4e8e-8564-45e956c10af7" (UID: "5fe90a87-f432-4e8e-8564-45e956c10af7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:23.453982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.453956 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-plugins-config-volume" pod "c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" (UID: "c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:23.460061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.460032 5824 generic.go:356] "Generic (PLEG): container finished" podID="c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" containerID="a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb" exitCode=0 Oct 30 14:50:23.460162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.460084 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" Oct 30 14:50:23.460162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.460112 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" event={"ID":"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7","Type":"ContainerDied","Data":"a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb"} Oct 30 14:50:23.460276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.460214 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7" event={"ID":"c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7","Type":"ContainerDied","Data":"e198d4123bb7ce1d54f43ce0b7f342ada736b50aa0d97312093a34f5e9766691"} Oct 30 14:50:23.460276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.460238 5824 scope.go:122] "RemoveContainer" containerID="a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb" Oct 30 14:50:23.462576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.461722 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" event={"ID":"3584ff8d-b5ca-413c-8abb-52ad766ae183","Type":"ContainerStarted","Data":"b5cfb791bbaaec5573688fe998bf2d0cff93b0d8760fabeafc5d7002bc36e149"} Oct 30 14:50:23.464521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.464213 5824 generic.go:356] "Generic (PLEG): container finished" podID="5fe90a87-f432-4e8e-8564-45e956c10af7" containerID="d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84" exitCode=0 Oct 30 14:50:23.464521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.464249 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" event={"ID":"5fe90a87-f432-4e8e-8564-45e956c10af7","Type":"ContainerDied","Data":"d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84"} Oct 30 14:50:23.464521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.464275 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" Oct 30 14:50:23.464521 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.464290 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw" event={"ID":"5fe90a87-f432-4e8e-8564-45e956c10af7","Type":"ContainerDied","Data":"cd663dccf4a6532b71b2a08cb15da337d71163678e0ab45ee79bd070a94211df"} Oct 30 14:50:23.465861 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.465837 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" event={"ID":"af496e84-e61e-48cc-9a60-234c05473f92","Type":"ContainerStarted","Data":"f8233c1ea4bd400141cf65b1247e662bff555e646bad61fc166af85ad70d1595"} Oct 30 14:50:23.470733 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.470715 5824 scope.go:122] "RemoveContainer" containerID="a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb" Oct 30 14:50:23.471023 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:23.471004 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb\": container with ID starting with a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb not found: ID does not exist" containerID="a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb" Oct 30 14:50:23.471074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.471031 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb"} err="failed to get container status \"a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb\": rpc error: code = NotFound desc = could not find container \"a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb\": container with ID starting with a72d4693c49d03d4cb590efb6b3b8e462946fba3c0fa46a009d31fb1f1c694cb not found: ID does not exist" Oct 30 14:50:23.471074 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.471048 5824 scope.go:122] "RemoveContainer" containerID="d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84" Oct 30 14:50:23.481924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.481904 5824 scope.go:122] "RemoveContainer" containerID="d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84" Oct 30 14:50:23.482222 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:23.482181 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84\": container with ID starting with d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84 not found: ID does not exist" containerID="d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84" Oct 30 14:50:23.482267 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.482233 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84"} err="failed to get container status \"d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84\": rpc error: code = NotFound desc = could not find container \"d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84\": container with ID starting with d3fdbd82d16d526b41acb655b745193f39f444771e07faa50758a7ba4e824d84 not found: ID does not exist" Oct 30 14:50:23.541883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.541825 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-kube-api-access-jr9mg" pod "c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" (UID: "c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7"). InnerVolumeSpecName "kube-api-access-jr9mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:23.541883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.541869 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe90a87-f432-4e8e-8564-45e956c10af7-kube-api-access-6g598" pod "5fe90a87-f432-4e8e-8564-45e956c10af7" (UID: "5fe90a87-f432-4e8e-8564-45e956c10af7"). InnerVolumeSpecName "kube-api-access-6g598". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:23.554258 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.554235 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g598\" (UniqueName: \"kubernetes.io/projected/5fe90a87-f432-4e8e-8564-45e956c10af7-kube-api-access-6g598\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:23.554357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.554260 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr9mg\" (UniqueName: \"kubernetes.io/projected/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-kube-api-access-jr9mg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:23.554357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.554274 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe90a87-f432-4e8e-8564-45e956c10af7-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:23.554357 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.554286 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:23.809229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.809134 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7"] Oct 30 14:50:23.819996 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.819956 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-228454-bcfd9db47-bzmp7"] Oct 30 14:50:23.862954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.862926 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw"] Oct 30 14:50:23.876016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:23.875988 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-228454-df8d85c7d-jzcbw"] Oct 30 14:50:24.317056 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:24.317032 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe90a87-f432-4e8e-8564-45e956c10af7" path="/var/lib/kubelet/pods/5fe90a87-f432-4e8e-8564-45e956c10af7/volumes" Oct 30 14:50:24.317451 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:24.317437 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a41048c-1202-4c97-b5a4-8a3f2c1e76bb" path="/var/lib/kubelet/pods/8a41048c-1202-4c97-b5a4-8a3f2c1e76bb/volumes" Oct 30 14:50:24.317767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:24.317755 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" path="/var/lib/kubelet/pods/c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7/volumes" Oct 30 14:50:24.475372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:24.475339 5824 generic.go:356] "Generic (PLEG): container finished" podID="af496e84-e61e-48cc-9a60-234c05473f92" containerID="bc0cfc5fdc752aee0089e5067f4e330833ce6f0b4942009fdbf2dbf48a0dda62" exitCode=0 Oct 30 14:50:24.475778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:24.475423 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" event={"ID":"af496e84-e61e-48cc-9a60-234c05473f92","Type":"ContainerDied","Data":"bc0cfc5fdc752aee0089e5067f4e330833ce6f0b4942009fdbf2dbf48a0dda62"} Oct 30 14:50:24.477729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:24.477704 5824 generic.go:356] "Generic (PLEG): container finished" podID="3584ff8d-b5ca-413c-8abb-52ad766ae183" containerID="0e79bcc8114bb8a9ed9dab3f9a8779550d2e4f6b1eaa9182a1c585abaf100aed" exitCode=0 Oct 30 14:50:24.477836 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:24.477757 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" event={"ID":"3584ff8d-b5ca-413c-8abb-52ad766ae183","Type":"ContainerDied","Data":"0e79bcc8114bb8a9ed9dab3f9a8779550d2e4f6b1eaa9182a1c585abaf100aed"} Oct 30 14:50:25.647607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.647582 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:25.666662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.666638 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:25.674456 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.674438 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-openshift-ca\") pod \"3584ff8d-b5ca-413c-8abb-52ad766ae183\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " Oct 30 14:50:25.674548 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.674527 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-xks-ca\") pod \"3584ff8d-b5ca-413c-8abb-52ad766ae183\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " Oct 30 14:50:25.674618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.674604 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3584ff8d-b5ca-413c-8abb-52ad766ae183-kube-api-access-6wm7l\" (UniqueName: \"kubernetes.io/projected/3584ff8d-b5ca-413c-8abb-52ad766ae183-kube-api-access-6wm7l\") pod \"3584ff8d-b5ca-413c-8abb-52ad766ae183\" (UID: \"3584ff8d-b5ca-413c-8abb-52ad766ae183\") " Oct 30 14:50:25.674754 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.674733 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-openshift-ca" pod "3584ff8d-b5ca-413c-8abb-52ad766ae183" (UID: "3584ff8d-b5ca-413c-8abb-52ad766ae183"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:25.674963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.674937 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:25.676537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.676519 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-xks-ca" pod "3584ff8d-b5ca-413c-8abb-52ad766ae183" (UID: "3584ff8d-b5ca-413c-8abb-52ad766ae183"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:25.676609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.676564 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3584ff8d-b5ca-413c-8abb-52ad766ae183-kube-api-access-6wm7l" pod "3584ff8d-b5ca-413c-8abb-52ad766ae183" (UID: "3584ff8d-b5ca-413c-8abb-52ad766ae183"). InnerVolumeSpecName "kube-api-access-6wm7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:25.776106 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.776018 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af496e84-e61e-48cc-9a60-234c05473f92-kube-api-access-7mdxt\" (UniqueName: \"kubernetes.io/projected/af496e84-e61e-48cc-9a60-234c05473f92-kube-api-access-7mdxt\") pod \"af496e84-e61e-48cc-9a60-234c05473f92\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " Oct 30 14:50:25.776331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.776103 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-openshift-ca\") pod \"af496e84-e61e-48cc-9a60-234c05473f92\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " Oct 30 14:50:25.776331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.776166 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-xks-ca\") pod \"af496e84-e61e-48cc-9a60-234c05473f92\" (UID: \"af496e84-e61e-48cc-9a60-234c05473f92\") " Oct 30 14:50:25.776465 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.776400 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-openshift-ca" pod "af496e84-e61e-48cc-9a60-234c05473f92" (UID: "af496e84-e61e-48cc-9a60-234c05473f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:25.776749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.776715 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584ff8d-b5ca-413c-8abb-52ad766ae183-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:25.776749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.776743 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wm7l\" (UniqueName: \"kubernetes.io/projected/3584ff8d-b5ca-413c-8abb-52ad766ae183-kube-api-access-6wm7l\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:25.776926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.776756 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:25.778310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.778275 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af496e84-e61e-48cc-9a60-234c05473f92-kube-api-access-7mdxt" pod "af496e84-e61e-48cc-9a60-234c05473f92" (UID: "af496e84-e61e-48cc-9a60-234c05473f92"). InnerVolumeSpecName "kube-api-access-7mdxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:25.778310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.778275 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-xks-ca" pod "af496e84-e61e-48cc-9a60-234c05473f92" (UID: "af496e84-e61e-48cc-9a60-234c05473f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:25.877652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.877629 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mdxt\" (UniqueName: \"kubernetes.io/projected/af496e84-e61e-48cc-9a60-234c05473f92-kube-api-access-7mdxt\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:25.877652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:25.877652 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af496e84-e61e-48cc-9a60-234c05473f92-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:26.493881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:26.493851 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" Oct 30 14:50:26.493881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:26.493867 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm" event={"ID":"3584ff8d-b5ca-413c-8abb-52ad766ae183","Type":"ContainerDied","Data":"b5cfb791bbaaec5573688fe998bf2d0cff93b0d8760fabeafc5d7002bc36e149"} Oct 30 14:50:26.494125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:26.493918 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5cfb791bbaaec5573688fe998bf2d0cff93b0d8760fabeafc5d7002bc36e149" Oct 30 14:50:26.495396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:26.495375 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" Oct 30 14:50:26.495529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:26.495425 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x" event={"ID":"af496e84-e61e-48cc-9a60-234c05473f92","Type":"ContainerDied","Data":"f8233c1ea4bd400141cf65b1247e662bff555e646bad61fc166af85ad70d1595"} Oct 30 14:50:26.495529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:26.495463 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8233c1ea4bd400141cf65b1247e662bff555e646bad61fc166af85ad70d1595" Oct 30 14:50:27.423708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.423670 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" podUID="6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b" containerName="payload-processing" containerID="cri-o://3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d" gracePeriod=25 Oct 30 14:50:27.456501 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.456474 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" podUID="f682f215-692e-4e8a-9ae2-76ea04c52ca3" containerName="payload-pre-processing" containerID="cri-o://e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f" gracePeriod=25 Oct 30 14:50:27.702622 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.702599 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:50:27.733061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.733040 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:50:27.794761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.794724 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-kube-api-access-r7fzl\" (UniqueName: \"kubernetes.io/projected/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-kube-api-access-r7fzl\") pod \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\" (UID: \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\") " Oct 30 14:50:27.794919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.794767 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f682f215-692e-4e8a-9ae2-76ea04c52ca3-kube-api-access-wrnjh\" (UniqueName: \"kubernetes.io/projected/f682f215-692e-4e8a-9ae2-76ea04c52ca3-kube-api-access-wrnjh\") pod \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\" (UID: \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\") " Oct 30 14:50:27.794919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.794798 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f682f215-692e-4e8a-9ae2-76ea04c52ca3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f682f215-692e-4e8a-9ae2-76ea04c52ca3-plugins-config-volume\") pod \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\" (UID: \"f682f215-692e-4e8a-9ae2-76ea04c52ca3\") " Oct 30 14:50:27.794919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.794834 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-plugins-config-volume\") pod \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\" (UID: \"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b\") " Oct 30 14:50:27.795152 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.795129 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f682f215-692e-4e8a-9ae2-76ea04c52ca3-plugins-config-volume" pod "f682f215-692e-4e8a-9ae2-76ea04c52ca3" (UID: "f682f215-692e-4e8a-9ae2-76ea04c52ca3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:27.795242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.795223 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-plugins-config-volume" pod "6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b" (UID: "6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:27.795301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.795284 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f682f215-692e-4e8a-9ae2-76ea04c52ca3-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:27.797071 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.797041 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-kube-api-access-r7fzl" pod "6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b" (UID: "6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b"). InnerVolumeSpecName "kube-api-access-r7fzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:27.797508 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.797480 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f682f215-692e-4e8a-9ae2-76ea04c52ca3-kube-api-access-wrnjh" pod "f682f215-692e-4e8a-9ae2-76ea04c52ca3" (UID: "f682f215-692e-4e8a-9ae2-76ea04c52ca3"). InnerVolumeSpecName "kube-api-access-wrnjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:27.896397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.896373 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7fzl\" (UniqueName: \"kubernetes.io/projected/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-kube-api-access-r7fzl\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:27.896514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.896400 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnjh\" (UniqueName: \"kubernetes.io/projected/f682f215-692e-4e8a-9ae2-76ea04c52ca3-kube-api-access-wrnjh\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:27.896514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:27.896413 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.042801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.042777 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:50:28.098418 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098388 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-credential-socket\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.098570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098436 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-data\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.098570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098506 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-certs\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.098570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098532 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-envoy\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.099236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098581 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad17e25-1d53-4cba-b2d4-bad295613862-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad17e25-1d53-4cba-b2d4-bad295613862-istiod-ca-cert\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.099236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098692 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-kube-api-access-5xkh8\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-kube-api-access-5xkh8\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.099236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098707 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-credential-socket" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:28.099236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098723 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-podinfo\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.099236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098770 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-socket\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.099236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098774 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-data" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:28.099236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098808 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-token\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-token\") pod \"9ad17e25-1d53-4cba-b2d4-bad295613862\" (UID: \"9ad17e25-1d53-4cba-b2d4-bad295613862\") " Oct 30 14:50:28.099236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.098805 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-certs" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:28.099706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.099382 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-socket" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:28.099706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.099439 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad17e25-1d53-4cba-b2d4-bad295613862-istiod-ca-cert" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:28.109213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.100236 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.109213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.100264 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.109213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.100283 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.109213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.100299 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.109213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.100319 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad17e25-1d53-4cba-b2d4-bad295613862-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.109213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.101696 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-envoy" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:28.109213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.101758 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-token" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:28.109213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.103139 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-podinfo" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:50:28.109977 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.109952 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-kube-api-access-5xkh8" pod "9ad17e25-1d53-4cba-b2d4-bad295613862" (UID: "9ad17e25-1d53-4cba-b2d4-bad295613862"). InnerVolumeSpecName "kube-api-access-5xkh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:28.201249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.201227 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.201249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.201248 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xkh8\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-kube-api-access-5xkh8\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.201388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.201258 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.201388 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.201266 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad17e25-1d53-4cba-b2d4-bad295613862-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:28.509960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.509931 5824 generic.go:356] "Generic (PLEG): container finished" podID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerID="761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3" exitCode=0 Oct 30 14:50:28.510346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.509989 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" Oct 30 14:50:28.510346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.510010 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" event={"ID":"9ad17e25-1d53-4cba-b2d4-bad295613862","Type":"ContainerDied","Data":"761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3"} Oct 30 14:50:28.510346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.510060 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t" event={"ID":"9ad17e25-1d53-4cba-b2d4-bad295613862","Type":"ContainerDied","Data":"710444635af5dc4609f9f87c2095d17569b7695b4ddb0f349bc4f174a90a357d"} Oct 30 14:50:28.510346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.510086 5824 scope.go:122] "RemoveContainer" containerID="761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3" Oct 30 14:50:28.511813 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.511775 5824 generic.go:356] "Generic (PLEG): container finished" podID="6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b" containerID="3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d" exitCode=0 Oct 30 14:50:28.511911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.511856 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" Oct 30 14:50:28.512013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.511992 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" event={"ID":"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b","Type":"ContainerDied","Data":"3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d"} Oct 30 14:50:28.512135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.512038 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4" event={"ID":"6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b","Type":"ContainerDied","Data":"77f46aecf87382f11bc6dc21fcf436ab8ef2faefa7e9e7684be0ce1fe67a8d5a"} Oct 30 14:50:28.513729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.513705 5824 generic.go:356] "Generic (PLEG): container finished" podID="f682f215-692e-4e8a-9ae2-76ea04c52ca3" containerID="e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f" exitCode=0 Oct 30 14:50:28.513847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.513763 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" event={"ID":"f682f215-692e-4e8a-9ae2-76ea04c52ca3","Type":"ContainerDied","Data":"e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f"} Oct 30 14:50:28.513847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.513802 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" event={"ID":"f682f215-692e-4e8a-9ae2-76ea04c52ca3","Type":"ContainerDied","Data":"a8d168a7a240a13f719544774a31293824af7a948555b39088cc89e13d65f8c3"} Oct 30 14:50:28.513847 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.513837 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z" Oct 30 14:50:28.520823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.520810 5824 scope.go:122] "RemoveContainer" containerID="912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3" Oct 30 14:50:28.529390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.529374 5824 scope.go:122] "RemoveContainer" containerID="761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3" Oct 30 14:50:28.529643 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:28.529623 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3\": container with ID starting with 761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3 not found: ID does not exist" containerID="761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3" Oct 30 14:50:28.529703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.529652 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3"} err="failed to get container status \"761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3\": rpc error: code = NotFound desc = could not find container \"761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3\": container with ID starting with 761e9bb0fbcd4530473a59fc9cd7533561a6810f2ffdf7b7d8dc130283177cd3 not found: ID does not exist" Oct 30 14:50:28.529703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.529669 5824 scope.go:122] "RemoveContainer" containerID="912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3" Oct 30 14:50:28.529929 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:28.529912 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3\": container with ID starting with 912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3 not found: ID does not exist" containerID="912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3" Oct 30 14:50:28.529971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.529934 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3"} err="failed to get container status \"912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3\": rpc error: code = NotFound desc = could not find container \"912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3\": container with ID starting with 912723d7edd89a694d60a4a4dad65497cdcb559e42560f1b83620ef2a08d2dc3 not found: ID does not exist" Oct 30 14:50:28.529971 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.529948 5824 scope.go:122] "RemoveContainer" containerID="3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d" Oct 30 14:50:28.540962 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.540941 5824 scope.go:122] "RemoveContainer" containerID="3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d" Oct 30 14:50:28.541237 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:28.541216 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d\": container with ID starting with 3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d not found: ID does not exist" containerID="3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d" Oct 30 14:50:28.541308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.541247 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d"} err="failed to get container status \"3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d\": rpc error: code = NotFound desc = could not find container \"3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d\": container with ID starting with 3f3dd7041d4f4fb82be03390f11629c8b95a14955c84b1bce427ad391a8bb24d not found: ID does not exist" Oct 30 14:50:28.541308 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.541264 5824 scope.go:122] "RemoveContainer" containerID="e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f" Oct 30 14:50:28.549329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.549314 5824 scope.go:122] "RemoveContainer" containerID="e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f" Oct 30 14:50:28.549580 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:28.549564 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f\": container with ID starting with e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f not found: ID does not exist" containerID="e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f" Oct 30 14:50:28.549625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.549584 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f"} err="failed to get container status \"e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f\": rpc error: code = NotFound desc = could not find container \"e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f\": container with ID starting with e67305b4816e03f1c29686f6e8fb0945889df4668a10b63e265590398960f70f not found: ID does not exist" Oct 30 14:50:28.567849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.567817 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4"] Oct 30 14:50:28.573872 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.573853 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-39529e-79cf96b4cb-qthk4"] Oct 30 14:50:28.613880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.613836 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t"] Oct 30 14:50:28.616803 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.616783 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2df0ab-openshift-default-757d5db88c-s268t"] Oct 30 14:50:28.655580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.655555 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z"] Oct 30 14:50:28.661356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:28.661334 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-39529e-7fcb96875-rx55z"] Oct 30 14:50:29.243963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:29.243931 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" podUID="81282091-cc28-4f1d-b6bd-aa68cadd63c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:50:30.322208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:30.320707 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b" path="/var/lib/kubelet/pods/6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b/volumes" Oct 30 14:50:30.322208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:30.321321 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" path="/var/lib/kubelet/pods/9ad17e25-1d53-4cba-b2d4-bad295613862/volumes" Oct 30 14:50:30.322208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:30.321702 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f682f215-692e-4e8a-9ae2-76ea04c52ca3" path="/var/lib/kubelet/pods/f682f215-692e-4e8a-9ae2-76ea04c52ca3/volumes" Oct 30 14:50:30.322208 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:30.322158 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n"] Oct 30 14:50:30.322833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:30.322358 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" containerID="cri-o://be70137b9f78d477f95d02adf4a2b7d04b00e94a390d0452e675e73701902003" gracePeriod=30 Oct 30 14:50:30.709471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:30.709428 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" containerID="cri-o://39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a" gracePeriod=45 Oct 30 14:50:30.957799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:30.957776 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:50:31.023664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.023592 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-model-cache\") pod \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " Oct 30 14:50:31.023664 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.023651 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tls-certs\") pod \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " Oct 30 14:50:31.023880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.023682 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tmp-dir\") pod \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " Oct 30 14:50:31.023880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.023759 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-home\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-home\") pod \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " Oct 30 14:50:31.023880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.023840 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-dshm\") pod \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " Oct 30 14:50:31.023880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.023871 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-kube-api-access-5nn68\" (UniqueName: \"kubernetes.io/projected/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-kube-api-access-5nn68\") pod \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\" (UID: \"13fb459c-d0d5-4498-a63c-0b8f4d288fdf\") " Oct 30 14:50:31.024120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.023885 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-model-cache" pod "13fb459c-d0d5-4498-a63c-0b8f4d288fdf" (UID: "13fb459c-d0d5-4498-a63c-0b8f4d288fdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.024120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.024030 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tmp-dir" pod "13fb459c-d0d5-4498-a63c-0b8f4d288fdf" (UID: "13fb459c-d0d5-4498-a63c-0b8f4d288fdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.024254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.024117 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-home" pod "13fb459c-d0d5-4498-a63c-0b8f4d288fdf" (UID: "13fb459c-d0d5-4498-a63c-0b8f4d288fdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.024311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.024300 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.024368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.024320 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.024368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.024337 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.026180 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.026158 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-dshm" pod "13fb459c-d0d5-4498-a63c-0b8f4d288fdf" (UID: "13fb459c-d0d5-4498-a63c-0b8f4d288fdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:31.026352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.026304 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-kube-api-access-5nn68" pod "13fb459c-d0d5-4498-a63c-0b8f4d288fdf" (UID: "13fb459c-d0d5-4498-a63c-0b8f4d288fdf"). InnerVolumeSpecName "kube-api-access-5nn68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:31.026352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.026328 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tls-certs" pod "13fb459c-d0d5-4498-a63c-0b8f4d288fdf" (UID: "13fb459c-d0d5-4498-a63c-0b8f4d288fdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:31.125015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.124982 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.125015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.125009 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.125015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.125018 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nn68\" (UniqueName: \"kubernetes.io/projected/13fb459c-d0d5-4498-a63c-0b8f4d288fdf-kube-api-access-5nn68\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:31.535389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.535357 5824 generic.go:356] "Generic (PLEG): container finished" podID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerID="39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a" exitCode=0 Oct 30 14:50:31.535794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.535405 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" Oct 30 14:50:31.535794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.535423 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" event={"ID":"13fb459c-d0d5-4498-a63c-0b8f4d288fdf","Type":"ContainerDied","Data":"39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a"} Oct 30 14:50:31.535794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.535454 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" event={"ID":"13fb459c-d0d5-4498-a63c-0b8f4d288fdf","Type":"ContainerDied","Data":"6f51aba32ce1d2fc0a2a02209d461d5389978e64c4e25ee67dde9c05455c4796"} Oct 30 14:50:31.535794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.535468 5824 scope.go:122] "RemoveContainer" containerID="39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a" Oct 30 14:50:31.546558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.546542 5824 scope.go:122] "RemoveContainer" containerID="39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a" Oct 30 14:50:31.546834 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:31.546813 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a\": container with ID starting with 39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a not found: ID does not exist" containerID="39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a" Oct 30 14:50:31.546902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.546846 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a"} err="failed to get container status \"39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a\": rpc error: code = NotFound desc = could not find container \"39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a\": container with ID starting with 39699dd800e2bd4fce4885ba0bd94c0699f791526f8c4b968eec8656227f0e1a not found: ID does not exist" Oct 30 14:50:31.592369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.592349 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh"] Oct 30 14:50:31.605607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.605588 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh"] Oct 30 14:50:31.915824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.915749 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00926c/premium-w3-00926c-kserve-75d5d7df67-8hhgh" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 14:50:31.948419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.948391 5824 patch_prober.go:29] interesting pod/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:50:31.948527 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:31.948444 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:50:32.315795 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:32.315770 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" path="/var/lib/kubelet/pods/13fb459c-d0d5-4498-a63c-0b8f4d288fdf/volumes" Oct 30 14:50:32.598424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:32.598365 5824 scope.go:122] "RemoveContainer" containerID="ec2120ab1545f223a4ed83e9ded93746b8826e32c8cf03b322896f0776b60ea7" Oct 30 14:50:33.549367 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.549331 5824 generic.go:356] "Generic (PLEG): container finished" podID="81282091-cc28-4f1d-b6bd-aa68cadd63c0" containerID="b43edd5ecf080e5a0b9b98f478e9592175e0ea2dba14966a05ba3e0204af12da" exitCode=0 Oct 30 14:50:33.549495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.549396 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" event={"ID":"81282091-cc28-4f1d-b6bd-aa68cadd63c0","Type":"ContainerDied","Data":"b43edd5ecf080e5a0b9b98f478e9592175e0ea2dba14966a05ba3e0204af12da"} Oct 30 14:50:33.549495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.549468 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" event={"ID":"81282091-cc28-4f1d-b6bd-aa68cadd63c0","Type":"ContainerDied","Data":"9aaf6ac8f33658d787bd5830fc80ae3273313d8e507d3b2dc0f78d7fd055487f"} Oct 30 14:50:33.549495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.549485 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aaf6ac8f33658d787bd5830fc80ae3273313d8e507d3b2dc0f78d7fd055487f" Oct 30 14:50:33.559522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.559505 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:50:33.639782 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:33.639698 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:50:33.650378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.650353 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81282091-cc28-4f1d-b6bd-aa68cadd63c0-kube-api-access-cqxlf\" (UniqueName: \"kubernetes.io/projected/81282091-cc28-4f1d-b6bd-aa68cadd63c0-kube-api-access-cqxlf\") pod \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\" (UID: \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\") " Oct 30 14:50:33.650528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.650410 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81282091-cc28-4f1d-b6bd-aa68cadd63c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81282091-cc28-4f1d-b6bd-aa68cadd63c0-plugins-config-volume\") pod \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\" (UID: \"81282091-cc28-4f1d-b6bd-aa68cadd63c0\") " Oct 30 14:50:33.650751 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.650730 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81282091-cc28-4f1d-b6bd-aa68cadd63c0-plugins-config-volume" pod "81282091-cc28-4f1d-b6bd-aa68cadd63c0" (UID: "81282091-cc28-4f1d-b6bd-aa68cadd63c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:33.652289 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.652265 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81282091-cc28-4f1d-b6bd-aa68cadd63c0-kube-api-access-cqxlf" pod "81282091-cc28-4f1d-b6bd-aa68cadd63c0" (UID: "81282091-cc28-4f1d-b6bd-aa68cadd63c0"). InnerVolumeSpecName "kube-api-access-cqxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:33.752078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.752051 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxlf\" (UniqueName: \"kubernetes.io/projected/81282091-cc28-4f1d-b6bd-aa68cadd63c0-kube-api-access-cqxlf\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:33.752078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:33.752075 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81282091-cc28-4f1d-b6bd-aa68cadd63c0-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:34.556087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:34.556056 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn" Oct 30 14:50:34.615776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:34.615711 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn"] Oct 30 14:50:34.629064 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:34.629039 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-46b13b-56878dd6bc-stcrn"] Oct 30 14:50:35.566381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.566274 5824 generic.go:356] "Generic (PLEG): container finished" podID="ff15b799-4e23-43b3-961d-c160bf18e272" containerID="be70137b9f78d477f95d02adf4a2b7d04b00e94a390d0452e675e73701902003" exitCode=0 Oct 30 14:50:35.566381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.566330 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" event={"ID":"ff15b799-4e23-43b3-961d-c160bf18e272","Type":"ContainerDied","Data":"be70137b9f78d477f95d02adf4a2b7d04b00e94a390d0452e675e73701902003"} Oct 30 14:50:35.566749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.566400 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" event={"ID":"ff15b799-4e23-43b3-961d-c160bf18e272","Type":"ContainerDied","Data":"a2d268c7d51391f7e29708d2042fd5aac1479dbcf89078dcc1b2d8f7be88a205"} Oct 30 14:50:35.566749 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.566427 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d268c7d51391f7e29708d2042fd5aac1479dbcf89078dcc1b2d8f7be88a205" Oct 30 14:50:35.579848 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.579818 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:50:35.665982 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.665908 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n"] Oct 30 14:50:35.668141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.668115 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00926c-wgs9n"] Oct 30 14:50:35.669968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.669946 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-socket\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.670076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.669997 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-istio-token\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-istio-token\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.670076 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670053 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-data\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.670310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670284 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-socket" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:35.670581 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670518 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff15b799-4e23-43b3-961d-c160bf18e272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff15b799-4e23-43b3-961d-c160bf18e272-istiod-ca-cert\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.670668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670600 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-data" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:35.670668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670629 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff15b799-4e23-43b3-961d-c160bf18e272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff15b799-4e23-43b3-961d-c160bf18e272-istio-podinfo\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.670785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670692 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-certs\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.670785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670734 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-envoy\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.670785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670765 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-kube-api-access-crghx\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-kube-api-access-crghx\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.670932 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.670853 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-credential-socket\") pod \"ff15b799-4e23-43b3-961d-c160bf18e272\" (UID: \"ff15b799-4e23-43b3-961d-c160bf18e272\") " Oct 30 14:50:35.671242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.671221 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.671320 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.671246 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.671447 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.671422 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-credential-socket" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:35.671596 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.671565 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-certs" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:35.671889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.671858 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff15b799-4e23-43b3-961d-c160bf18e272-istiod-ca-cert" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:35.673509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.673488 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff15b799-4e23-43b3-961d-c160bf18e272-istio-podinfo" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:50:35.673617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.673536 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-istio-token" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:35.673617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.673564 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-kube-api-access-crghx" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "kube-api-access-crghx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:35.674463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.674435 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-envoy" pod "ff15b799-4e23-43b3-961d-c160bf18e272" (UID: "ff15b799-4e23-43b3-961d-c160bf18e272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:35.771898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.771857 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff15b799-4e23-43b3-961d-c160bf18e272-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.771898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.771889 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.771898 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.771903 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.772139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.771914 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crghx\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-kube-api-access-crghx\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.772139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.771928 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff15b799-4e23-43b3-961d-c160bf18e272-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.772139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.771942 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff15b799-4e23-43b3-961d-c160bf18e272-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:35.772139 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:35.771953 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff15b799-4e23-43b3-961d-c160bf18e272-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:36.315471 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:36.315440 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81282091-cc28-4f1d-b6bd-aa68cadd63c0" path="/var/lib/kubelet/pods/81282091-cc28-4f1d-b6bd-aa68cadd63c0/volumes" Oct 30 14:50:36.315849 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:36.315834 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa" path="/var/lib/kubelet/pods/9de6e34c-e6fd-4ffd-8a8e-7110fc7d44aa/volumes" Oct 30 14:50:36.572070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:36.572000 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n" Oct 30 14:50:36.643791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:36.643767 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n"] Oct 30 14:50:36.658141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:36.658110 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-46b13b-openshift-default-7b98ffbf7c-b4j8n"] Oct 30 14:50:38.314454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:38.314422 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" path="/var/lib/kubelet/pods/ff15b799-4e23-43b3-961d-c160bf18e272/volumes" Oct 30 14:50:44.823878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:44.823834 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd"] Oct 30 14:50:44.824390 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:44.824054 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" containerID="cri-o://db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab" gracePeriod=30 Oct 30 14:50:46.675609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:46.675573 5824 patch_prober.go:29] interesting pod/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 14:50:46.675975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:46.675634 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 14:50:50.067607 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.067578 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:50:50.210247 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210219 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-token\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-token\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210271 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-podinfo\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210373 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-kube-api-access-zx8mp\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-kube-api-access-zx8mp\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210413 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-data\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210449 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-socket\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210485 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-credential-socket\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210516 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-certs\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210553 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-envoy\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210583 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istiod-ca-cert\") pod \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\" (UID: \"d47280eb-1f64-4558-99bf-21d0ed2a41d1\") " Oct 30 14:50:50.210832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210705 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-data" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:50.210969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210935 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-socket" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:50.210969 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.210959 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.211105 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.211029 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istiod-ca-cert" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:50:50.211221 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.211170 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-certs" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:50.211279 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.211264 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-credential-socket" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:50.212537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.212508 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-token" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:50.212656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.212602 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-podinfo" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:50:50.212656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.212603 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-kube-api-access-zx8mp" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "kube-api-access-zx8mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:50.212889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.212869 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-envoy" pod "d47280eb-1f64-4558-99bf-21d0ed2a41d1" (UID: "d47280eb-1f64-4558-99bf-21d0ed2a41d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:50.312084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.312061 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8mp\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-kube-api-access-zx8mp\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.312178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.312090 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.312178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.312106 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.312178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.312119 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.312178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.312133 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.312178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.312145 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.312178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.312157 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.312178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.312169 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d47280eb-1f64-4558-99bf-21d0ed2a41d1-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:50.653644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.653581 5824 generic.go:356] "Generic (PLEG): container finished" podID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerID="db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab" exitCode=0 Oct 30 14:50:50.653644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.653626 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" Oct 30 14:50:50.653814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.653662 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" event={"ID":"d47280eb-1f64-4558-99bf-21d0ed2a41d1","Type":"ContainerDied","Data":"db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab"} Oct 30 14:50:50.653814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.653714 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd" event={"ID":"d47280eb-1f64-4558-99bf-21d0ed2a41d1","Type":"ContainerDied","Data":"02ac105c73db5e37e896714fc6435ec7fbba1c9caee4ef0d03f75cad2361b24e"} Oct 30 14:50:50.653814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.653736 5824 scope.go:122] "RemoveContainer" containerID="db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab" Oct 30 14:50:50.665627 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.665611 5824 scope.go:122] "RemoveContainer" containerID="db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab" Oct 30 14:50:50.665909 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:50:50.665880 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab\": container with ID starting with db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab not found: ID does not exist" containerID="db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab" Oct 30 14:50:50.665975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.665913 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab"} err="failed to get container status \"db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab\": rpc error: code = NotFound desc = could not find container \"db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab\": container with ID starting with db963278486fb6c2765e1b42995293448589c08da29c01db09071708899357ab not found: ID does not exist" Oct 30 14:50:50.718013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.717992 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd"] Oct 30 14:50:50.731562 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:50.731539 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-39529e-openshift-default-cb57f9db7-m75qd"] Oct 30 14:50:52.315338 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:52.315306 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" path="/var/lib/kubelet/pods/d47280eb-1f64-4558-99bf-21d0ed2a41d1/volumes" Oct 30 14:50:59.616225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.616178 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75db85b96b-k2lg7"] Oct 30 14:50:59.616717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.616397 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75db85b96b-k2lg7" podUID="1c0be2a8-6c76-45c9-b301-4e40d5aad04d" containerName="manager" containerID="cri-o://5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1" gracePeriod=30 Oct 30 14:50:59.863045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.863018 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:50:59.898131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.898080 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-kube-api-access-nvglv\" (UniqueName: \"kubernetes.io/projected/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-kube-api-access-nvglv\") pod \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " Oct 30 14:50:59.898261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.898140 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-metrics-certs\") pod \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " Oct 30 14:50:59.898261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.898202 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-webhook-cert\") pod \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\" (UID: \"1c0be2a8-6c76-45c9-b301-4e40d5aad04d\") " Oct 30 14:50:59.900065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.900041 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-webhook-cert" pod "1c0be2a8-6c76-45c9-b301-4e40d5aad04d" (UID: "1c0be2a8-6c76-45c9-b301-4e40d5aad04d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:59.900159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.900063 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-kube-api-access-nvglv" pod "1c0be2a8-6c76-45c9-b301-4e40d5aad04d" (UID: "1c0be2a8-6c76-45c9-b301-4e40d5aad04d"). InnerVolumeSpecName "kube-api-access-nvglv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:59.900159 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.900067 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-metrics-certs" pod "1c0be2a8-6c76-45c9-b301-4e40d5aad04d" (UID: "1c0be2a8-6c76-45c9-b301-4e40d5aad04d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:50:59.998930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.998895 5824 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-webhook-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:59.998930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.998924 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvglv\" (UniqueName: \"kubernetes.io/projected/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-kube-api-access-nvglv\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:50:59.998930 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:50:59.998937 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0be2a8-6c76-45c9-b301-4e40d5aad04d-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:00.713244 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.713216 5824 generic.go:356] "Generic (PLEG): container finished" podID="1c0be2a8-6c76-45c9-b301-4e40d5aad04d" containerID="5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1" exitCode=0 Oct 30 14:51:00.713604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.713312 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-k2lg7" event={"ID":"1c0be2a8-6c76-45c9-b301-4e40d5aad04d","Type":"ContainerDied","Data":"5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1"} Oct 30 14:51:00.713604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.713338 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75db85b96b-k2lg7" Oct 30 14:51:00.713604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.713378 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75db85b96b-k2lg7" event={"ID":"1c0be2a8-6c76-45c9-b301-4e40d5aad04d","Type":"ContainerDied","Data":"f9109701e3b983d7d2108818077eb0dc03b1fff68c93f3d83624731b4e36029f"} Oct 30 14:51:00.713604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.713401 5824 scope.go:122] "RemoveContainer" containerID="5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1" Oct 30 14:51:00.723673 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.723653 5824 scope.go:122] "RemoveContainer" containerID="5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1" Oct 30 14:51:00.723951 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:51:00.723930 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1\": container with ID starting with 5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1 not found: ID does not exist" containerID="5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1" Oct 30 14:51:00.724030 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.723957 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1"} err="failed to get container status \"5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1\": rpc error: code = NotFound desc = could not find container \"5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1\": container with ID starting with 5724214118f62b9fdbef4412297f971442adeb29869ea84863293424f0e35ad1 not found: ID does not exist" Oct 30 14:51:00.792864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.792840 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75db85b96b-k2lg7"] Oct 30 14:51:00.812544 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:00.812513 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75db85b96b-k2lg7"] Oct 30 14:51:02.314551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:02.314517 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0be2a8-6c76-45c9-b301-4e40d5aad04d" path="/var/lib/kubelet/pods/1c0be2a8-6c76-45c9-b301-4e40d5aad04d/volumes" Oct 30 14:51:03.640974 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:51:03.640940 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:51:04.895572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:04.895542 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg"] Oct 30 14:51:04.895990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:04.895727 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" podUID="97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" containerName="discovery" containerID="cri-o://1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf" gracePeriod=30 Oct 30 14:51:05.141568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.141544 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:51:05.242262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.242232 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-cacerts\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-cacerts\") pod \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " Oct 30 14:51:05.242427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.242267 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-kubeconfig\") pod \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " Oct 30 14:51:05.242427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.242296 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-kube-api-access-bqrq4\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-kube-api-access-bqrq4\") pod \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " Oct 30 14:51:05.242427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.242335 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-local-certs\") pod \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " Oct 30 14:51:05.242427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.242373 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-ca-configmap\") pod \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " Oct 30 14:51:05.242644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.242463 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-dns-cert\") pod \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " Oct 30 14:51:05.242644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.242495 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-token\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-token\") pod \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\" (UID: \"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4\") " Oct 30 14:51:05.242862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.242831 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-ca-configmap" pod "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" (UID: "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:51:05.244618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.244585 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-kube-api-access-bqrq4" pod "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" (UID: "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4"). InnerVolumeSpecName "kube-api-access-bqrq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:51:05.244734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.244614 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-kubeconfig" pod "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" (UID: "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:51:05.244823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.244801 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-cacerts" pod "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" (UID: "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:51:05.245120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.245098 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-token" pod "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" (UID: "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:51:05.245185 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.245110 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-dns-cert" pod "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" (UID: "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:51:05.245185 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.245122 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-local-certs" pod "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" (UID: "97d5e1f9-6e09-4021-a6f7-9a3546efb0c4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:51:05.343989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.343911 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:05.343989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.343933 5824 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-cacerts\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:05.343989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.343941 5824 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-kubeconfig\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:05.343989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.343949 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqrq4\" (UniqueName: \"kubernetes.io/projected/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-kube-api-access-bqrq4\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:05.343989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.343958 5824 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-local-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:05.343989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.343965 5824 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-ca-configmap\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:05.343989 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.343973 5824 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4-istio-csr-dns-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:05.743635 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.743598 5824 generic.go:356] "Generic (PLEG): container finished" podID="97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" containerID="1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf" exitCode=0 Oct 30 14:51:05.743812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.743667 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" Oct 30 14:51:05.743812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.743684 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" event={"ID":"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4","Type":"ContainerDied","Data":"1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf"} Oct 30 14:51:05.743812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.743768 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg" event={"ID":"97d5e1f9-6e09-4021-a6f7-9a3546efb0c4","Type":"ContainerDied","Data":"ac8301e406212f1fd54f024e9fbf7e081366492ba95a1440f1b0353ac253b37e"} Oct 30 14:51:05.743812 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.743796 5824 scope.go:122] "RemoveContainer" containerID="1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf" Oct 30 14:51:05.754385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.754322 5824 scope.go:122] "RemoveContainer" containerID="1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf" Oct 30 14:51:05.754614 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:51:05.754594 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf\": container with ID starting with 1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf not found: ID does not exist" containerID="1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf" Oct 30 14:51:05.754667 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.754622 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf"} err="failed to get container status \"1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf\": rpc error: code = NotFound desc = could not find container \"1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf\": container with ID starting with 1937ff41592def6707507100d2282b8b0316540989e248606749b7af579dfcbf not found: ID does not exist" Oct 30 14:51:05.801674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.801653 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg"] Oct 30 14:51:05.814981 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:05.814960 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-68bcg"] Oct 30 14:51:06.315911 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:06.315878 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" path="/var/lib/kubelet/pods/97d5e1f9-6e09-4021-a6f7-9a3546efb0c4/volumes" Oct 30 14:51:11.403575 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403498 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx"] Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403850 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1c0be2a8-6c76-45c9-b301-4e40d5aad04d" containerName="manager" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403861 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0be2a8-6c76-45c9-b301-4e40d5aad04d" containerName="manager" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403869 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403874 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403882 5824 cpu_manager.go:414] "containerMap: removing container" podUID="f682f215-692e-4e8a-9ae2-76ea04c52ca3" containerName="payload-pre-processing" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403887 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f682f215-692e-4e8a-9ae2-76ea04c52ca3" containerName="payload-pre-processing" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403903 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" containerName="payload-pre-processing" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403908 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" containerName="payload-pre-processing" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403916 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5fe90a87-f432-4e8e-8564-45e956c10af7" containerName="payload-processing" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403921 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe90a87-f432-4e8e-8564-45e956c10af7" containerName="payload-processing" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403933 5824 cpu_manager.go:414] "containerMap: removing container" podUID="97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" containerName="discovery" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403937 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" containerName="discovery" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403949 5824 cpu_manager.go:414] "containerMap: removing container" podUID="81282091-cc28-4f1d-b6bd-aa68cadd63c0" containerName="payload-pre-processing" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403954 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81282091-cc28-4f1d-b6bd-aa68cadd63c0" containerName="payload-pre-processing" Oct 30 14:51:11.403951 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403960 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403964 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403977 5824 cpu_manager.go:414] "containerMap: removing container" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403981 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403992 5824 cpu_manager.go:414] "containerMap: removing container" podUID="3584ff8d-b5ca-413c-8abb-52ad766ae183" containerName="revoke-keys" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.403996 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3584ff8d-b5ca-413c-8abb-52ad766ae183" containerName="revoke-keys" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404002 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b" containerName="payload-processing" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404006 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b" containerName="payload-processing" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404013 5824 cpu_manager.go:414] "containerMap: removing container" podUID="af496e84-e61e-48cc-9a60-234c05473f92" containerName="revoke-keys" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404017 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af496e84-e61e-48cc-9a60-234c05473f92" containerName="revoke-keys" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404023 5824 cpu_manager.go:414] "containerMap: removing container" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404027 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404078 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04ffdb6-b2dc-4cd6-95b3-d860e3f648f7" containerName="payload-pre-processing" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404087 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe90a87-f432-4e8e-8564-45e956c10af7" containerName="payload-processing" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404094 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d5e1f9-6e09-4021-a6f7-9a3546efb0c4" containerName="discovery" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404100 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404105 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="81282091-cc28-4f1d-b6bd-aa68cadd63c0" containerName="payload-pre-processing" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404110 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff15b799-4e23-43b3-961d-c160bf18e272" containerName="istio-proxy" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404117 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47280eb-1f64-4558-99bf-21d0ed2a41d1" containerName="istio-proxy" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404122 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="3584ff8d-b5ca-413c-8abb-52ad766ae183" containerName="revoke-keys" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404127 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cac6cc1-b6eb-4ef5-adf8-9c0eac9f393b" containerName="payload-processing" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404133 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="af496e84-e61e-48cc-9a60-234c05473f92" containerName="revoke-keys" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404137 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fb459c-d0d5-4498-a63c-0b8f4d288fdf" containerName="main" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404142 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0be2a8-6c76-45c9-b301-4e40d5aad04d" containerName="manager" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404146 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" Oct 30 14:51:11.404601 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.404151 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="f682f215-692e-4e8a-9ae2-76ea04c52ca3" containerName="payload-pre-processing" Oct 30 14:51:11.408816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.408798 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.437510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.437489 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 14:51:11.439168 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.439139 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx"] Oct 30 14:51:11.489585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489559 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.489700 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489597 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc4102f-81bc-4831-bf65-fcd79c204148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.489700 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489626 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.489700 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489667 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.489815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489733 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.489815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489760 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.489815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489784 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.489935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489856 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6btrm\" (UniqueName: \"kubernetes.io/projected/6bc4102f-81bc-4831-bf65-fcd79c204148-kube-api-access-6btrm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.489935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.489915 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.518311 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.518289 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5drtm\"" Oct 30 14:51:11.590726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590702 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.590726 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590726 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.590863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590745 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.590863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590763 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6btrm\" (UniqueName: \"kubernetes.io/projected/6bc4102f-81bc-4831-bf65-fcd79c204148-kube-api-access-6btrm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.590863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590804 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.590863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590837 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.590863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590860 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc4102f-81bc-4831-bf65-fcd79c204148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.591223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590898 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.591223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.590924 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.591346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.591309 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.591346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.591332 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.591499 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.591477 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.591564 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.591528 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.591767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.591752 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc4102f-81bc-4831-bf65-fcd79c204148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.593539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.593521 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.593668 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.593652 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.616878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.616843 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc4102f-81bc-4831-bf65-fcd79c204148-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.617022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.616994 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6btrm\" (UniqueName: \"kubernetes.io/projected/6bc4102f-81bc-4831-bf65-fcd79c204148-kube-api-access-6btrm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cl2qx\" (UID: \"6bc4102f-81bc-4831-bf65-fcd79c204148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:11.721914 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:11.721890 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:12.035807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:12.035779 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx"] Oct 30 14:51:12.036508 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:51:12.036488 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc4102f_81bc_4831_bf65_fcd79c204148.slice/crio-03000533c26af83d9719a953f130ed2e46e3000d25332ec3d8bf4b60b0ea7ae0 WatchSource:0}: Error finding container 03000533c26af83d9719a953f130ed2e46e3000d25332ec3d8bf4b60b0ea7ae0: Status 404 returned error can't find the container with id 03000533c26af83d9719a953f130ed2e46e3000d25332ec3d8bf4b60b0ea7ae0 Oct 30 14:51:12.038484 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:12.038447 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:51:12.038551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:12.038516 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:51:12.038551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:12.038545 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:51:12.794663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:12.794573 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" event={"ID":"6bc4102f-81bc-4831-bf65-fcd79c204148","Type":"ContainerStarted","Data":"a0ad9dc152e7eb61f8e4062a9bdd6aa3e22d5400b5601c56cfdb9bdbf9ee2fc6"} Oct 30 14:51:12.794663 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:12.794634 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" event={"ID":"6bc4102f-81bc-4831-bf65-fcd79c204148","Type":"ContainerStarted","Data":"03000533c26af83d9719a953f130ed2e46e3000d25332ec3d8bf4b60b0ea7ae0"} Oct 30 14:51:12.865213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:12.863498 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" podStartSLOduration=1.86347843 podStartE2EDuration="1.86347843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:12.862303355 +0000 UTC m=+3702.662808800" watchObservedRunningTime="2026-10-30 14:51:12.86347843 +0000 UTC m=+3702.663983874" Oct 30 14:51:13.722349 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:13.722319 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:13.723801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:13.723772 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 14:51:13.723945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:13.723838 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" podUID="6bc4102f-81bc-4831-bf65-fcd79c204148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 14:51:14.722545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:14.722509 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 14:51:14.722995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:14.722577 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" podUID="6bc4102f-81bc-4831-bf65-fcd79c204148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 14:51:15.722644 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:15.722607 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 14:51:15.723053 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:15.722659 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" podUID="6bc4102f-81bc-4831-bf65-fcd79c204148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 14:51:16.722835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:16.722800 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 14:51:16.723346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:16.722867 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" podUID="6bc4102f-81bc-4831-bf65-fcd79c204148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 14:51:17.722536 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:17.722502 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 14:51:17.722718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:17.722560 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" podUID="6bc4102f-81bc-4831-bf65-fcd79c204148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 14:51:18.723121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:18.723082 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 14:51:18.723592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:18.723143 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" podUID="6bc4102f-81bc-4831-bf65-fcd79c204148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 14:51:19.970178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:19.970145 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:19.970640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:19.970607 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:19.971352 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:19.971329 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" Oct 30 14:51:24.747678 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.747647 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f"] Oct 30 14:51:24.748081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.747996 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" Oct 30 14:51:24.748081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.748007 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad17e25-1d53-4cba-b2d4-bad295613862" containerName="istio-proxy" Oct 30 14:51:24.751389 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.751373 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.769590 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.769567 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f"] Oct 30 14:51:24.781132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.781106 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:51:24.781132 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.781111 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:51:24.781322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.781185 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:51:24.781322 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.781263 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:51:24.836248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.836225 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.836380 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.836262 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ddh\" (UniqueName: \"kubernetes.io/projected/33b982f2-9323-4aec-83a5-63ecaf23463c-kube-api-access-n7ddh\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.836380 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.836304 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.937230 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.937182 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.937325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.937231 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ddh\" (UniqueName: \"kubernetes.io/projected/33b982f2-9323-4aec-83a5-63ecaf23463c-kube-api-access-n7ddh\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.937325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.937256 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.937863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.937837 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.939416 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.939391 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:24.964295 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:24.964272 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ddh\" (UniqueName: \"kubernetes.io/projected/33b982f2-9323-4aec-83a5-63ecaf23463c-kube-api-access-n7ddh\") pod \"maas-api-revoke-sub-85237e45-8c52f\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:25.064591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:25.064547 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:25.360892 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:25.360866 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f"] Oct 30 14:51:25.362384 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:51:25.362354 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b982f2_9323_4aec_83a5_63ecaf23463c.slice/crio-904fdb4ce7785acf614af8fc2771776dce401d2df0b78c4cb3efb0eeb41e252d WatchSource:0}: Error finding container 904fdb4ce7785acf614af8fc2771776dce401d2df0b78c4cb3efb0eeb41e252d: Status 404 returned error can't find the container with id 904fdb4ce7785acf614af8fc2771776dce401d2df0b78c4cb3efb0eeb41e252d Oct 30 14:51:25.873103 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:25.873077 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" event={"ID":"33b982f2-9323-4aec-83a5-63ecaf23463c","Type":"ContainerStarted","Data":"a439b1b813d2e1f75db56217c95412e2d68bab14843434d609a98f8a74f4e403"} Oct 30 14:51:25.873432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:25.873127 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" event={"ID":"33b982f2-9323-4aec-83a5-63ecaf23463c","Type":"ContainerStarted","Data":"904fdb4ce7785acf614af8fc2771776dce401d2df0b78c4cb3efb0eeb41e252d"} Oct 30 14:51:25.917700 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:25.917655 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" podStartSLOduration=1.91764174 podStartE2EDuration="1.91764174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:25.916416305 +0000 UTC m=+3715.716921747" watchObservedRunningTime="2026-10-30 14:51:25.91764174 +0000 UTC m=+3715.718147249" Oct 30 14:51:26.437404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.437367 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b"] Oct 30 14:51:26.441095 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.441067 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.474038 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.474005 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b"] Oct 30 14:51:26.551079 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551044 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsl4s\" (UniqueName: \"kubernetes.io/projected/d5057b65-74d7-4c84-b500-ba697b16459f-kube-api-access-dsl4s\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.551261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551086 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.551261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551167 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5057b65-74d7-4c84-b500-ba697b16459f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.551261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551232 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.551261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551258 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5057b65-74d7-4c84-b500-ba697b16459f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.551443 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551365 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.551492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551443 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.551492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551471 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.551596 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.551516 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5057b65-74d7-4c84-b500-ba697b16459f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.652918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.652894 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsl4s\" (UniqueName: \"kubernetes.io/projected/d5057b65-74d7-4c84-b500-ba697b16459f-kube-api-access-dsl4s\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.652944 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.652982 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5057b65-74d7-4c84-b500-ba697b16459f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653013 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653036 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5057b65-74d7-4c84-b500-ba697b16459f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653083 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653133 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653174 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653333 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653243 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5057b65-74d7-4c84-b500-ba697b16459f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653509 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653718 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653692 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653849 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.653899 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.653873 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.654059 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.654033 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5057b65-74d7-4c84-b500-ba697b16459f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.655301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.655279 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5057b65-74d7-4c84-b500-ba697b16459f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.655633 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.655613 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5057b65-74d7-4c84-b500-ba697b16459f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.678608 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.678582 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5057b65-74d7-4c84-b500-ba697b16459f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.678740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.678720 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsl4s\" (UniqueName: \"kubernetes.io/projected/d5057b65-74d7-4c84-b500-ba697b16459f-kube-api-access-dsl4s\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gqq8b\" (UID: \"d5057b65-74d7-4c84-b500-ba697b16459f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.754184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.754134 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:26.883083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.883049 5824 generic.go:356] "Generic (PLEG): container finished" podID="33b982f2-9323-4aec-83a5-63ecaf23463c" containerID="a439b1b813d2e1f75db56217c95412e2d68bab14843434d609a98f8a74f4e403" exitCode=0 Oct 30 14:51:26.883474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:26.883137 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" event={"ID":"33b982f2-9323-4aec-83a5-63ecaf23463c","Type":"ContainerDied","Data":"a439b1b813d2e1f75db56217c95412e2d68bab14843434d609a98f8a74f4e403"} Oct 30 14:51:27.075243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:27.075212 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b"] Oct 30 14:51:27.075797 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:51:27.075770 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5057b65_74d7_4c84_b500_ba697b16459f.slice/crio-372e0743ce192a5af7cbcc21a74ac9958a796342a8e97de8b50971f87c1e68f3 WatchSource:0}: Error finding container 372e0743ce192a5af7cbcc21a74ac9958a796342a8e97de8b50971f87c1e68f3: Status 404 returned error can't find the container with id 372e0743ce192a5af7cbcc21a74ac9958a796342a8e97de8b50971f87c1e68f3 Oct 30 14:51:27.078044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:27.078008 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:51:27.078129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:27.078076 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:51:27.078129 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:27.078102 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:51:27.899255 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:27.899214 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" event={"ID":"d5057b65-74d7-4c84-b500-ba697b16459f","Type":"ContainerStarted","Data":"bfb0d5021d9bc69a075c7cf4cdc76ac7e33b3d0d4e3c56b411dee1fa9928c21a"} Oct 30 14:51:27.900366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:27.900328 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" event={"ID":"d5057b65-74d7-4c84-b500-ba697b16459f","Type":"ContainerStarted","Data":"372e0743ce192a5af7cbcc21a74ac9958a796342a8e97de8b50971f87c1e68f3"} Oct 30 14:51:27.985213 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:27.966531 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" podStartSLOduration=1.9665096869999998 podStartE2EDuration="1.966509687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:27.964688366 +0000 UTC m=+3717.765193812" watchObservedRunningTime="2026-10-30 14:51:27.966509687 +0000 UTC m=+3717.767015132" Oct 30 14:51:28.112519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.112491 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:28.193218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.192630 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-xks-ca\") pod \"33b982f2-9323-4aec-83a5-63ecaf23463c\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " Oct 30 14:51:28.193218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.192771 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-openshift-ca\") pod \"33b982f2-9323-4aec-83a5-63ecaf23463c\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " Oct 30 14:51:28.193218 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.192901 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b982f2-9323-4aec-83a5-63ecaf23463c-kube-api-access-n7ddh\" (UniqueName: \"kubernetes.io/projected/33b982f2-9323-4aec-83a5-63ecaf23463c-kube-api-access-n7ddh\") pod \"33b982f2-9323-4aec-83a5-63ecaf23463c\" (UID: \"33b982f2-9323-4aec-83a5-63ecaf23463c\") " Oct 30 14:51:28.196233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.193467 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-openshift-ca" pod "33b982f2-9323-4aec-83a5-63ecaf23463c" (UID: "33b982f2-9323-4aec-83a5-63ecaf23463c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:51:28.196233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.193807 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:28.196233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.195789 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b982f2-9323-4aec-83a5-63ecaf23463c-kube-api-access-n7ddh" pod "33b982f2-9323-4aec-83a5-63ecaf23463c" (UID: "33b982f2-9323-4aec-83a5-63ecaf23463c"). InnerVolumeSpecName "kube-api-access-n7ddh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:51:28.198618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.198578 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-xks-ca" pod "33b982f2-9323-4aec-83a5-63ecaf23463c" (UID: "33b982f2-9323-4aec-83a5-63ecaf23463c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:51:28.295224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.295161 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7ddh\" (UniqueName: \"kubernetes.io/projected/33b982f2-9323-4aec-83a5-63ecaf23463c-kube-api-access-n7ddh\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:28.295224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.295225 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b982f2-9323-4aec-83a5-63ecaf23463c-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:51:28.762271 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.762171 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:28.766746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.766383 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:51:28.766746 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.766441 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" podUID="d5057b65-74d7-4c84-b500-ba697b16459f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 14:51:28.906864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.906833 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" Oct 30 14:51:28.907310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.906864 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f" event={"ID":"33b982f2-9323-4aec-83a5-63ecaf23463c","Type":"ContainerDied","Data":"904fdb4ce7785acf614af8fc2771776dce401d2df0b78c4cb3efb0eeb41e252d"} Oct 30 14:51:28.907310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:28.906908 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="904fdb4ce7785acf614af8fc2771776dce401d2df0b78c4cb3efb0eeb41e252d" Oct 30 14:51:29.766332 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:29.766289 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:51:29.766584 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:29.766348 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" podUID="d5057b65-74d7-4c84-b500-ba697b16459f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 14:51:30.760214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:30.756470 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:51:30.760214 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:30.756532 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" podUID="d5057b65-74d7-4c84-b500-ba697b16459f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 14:51:31.757495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:31.757459 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:51:31.757777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:31.757749 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" podUID="d5057b65-74d7-4c84-b500-ba697b16459f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 14:51:32.732532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:32.732502 5824 scope.go:122] "RemoveContainer" containerID="b43edd5ecf080e5a0b9b98f478e9592175e0ea2dba14966a05ba3e0204af12da" Oct 30 14:51:32.751974 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:32.751276 5824 scope.go:122] "RemoveContainer" containerID="a5a47bfb9dcf3c5581ce6e41fe0c322f9a4f0323eca8a19c0d07438de0ea2a32" Oct 30 14:51:32.755369 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:32.755339 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:51:32.755482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:32.755394 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" podUID="d5057b65-74d7-4c84-b500-ba697b16459f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 14:51:32.763264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:32.763244 5824 scope.go:122] "RemoveContainer" containerID="be70137b9f78d477f95d02adf4a2b7d04b00e94a390d0452e675e73701902003" Oct 30 14:51:32.771599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:32.771571 5824 scope.go:122] "RemoveContainer" containerID="6f68355cd97ff88c056cb12c45f49ef90c8eee8ea12cfc6234ae0af8ae026c51" Oct 30 14:51:33.638665 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:51:33.638627 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:51:33.755341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:33.755310 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Oct 30 14:51:33.755789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:33.755363 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" podUID="d5057b65-74d7-4c84-b500-ba697b16459f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Oct 30 14:51:35.097248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:35.097216 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:35.097725 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:35.097503 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:35.098291 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:35.098273 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" Oct 30 14:51:51.064654 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:51.064603 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht"] Oct 30 14:51:51.066325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:51.066298 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx"] Oct 30 14:51:51.067643 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:51.067626 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-908d8733-gtbht"] Oct 30 14:51:51.068880 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:51.068856 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5b2199-l9fvx"] Oct 30 14:51:52.314891 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:52.314858 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7534860f-19f0-4be5-8e90-a9b351094f54" path="/var/lib/kubelet/pods/7534860f-19f0-4be5-8e90-a9b351094f54/volumes" Oct 30 14:51:52.315302 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:52.315267 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cafb40-d317-4d4e-bf1d-b72650046fe1" path="/var/lib/kubelet/pods/f4cafb40-d317-4d4e-bf1d-b72650046fe1/volumes" Oct 30 14:51:55.056666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:55.056632 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv"] Oct 30 14:51:55.070209 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:55.070161 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcacc078-8z4sv"] Oct 30 14:51:56.315786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:51:56.315747 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd56fe2-85ca-473d-8db8-3ed75acd5c65" path="/var/lib/kubelet/pods/7bd56fe2-85ca-473d-8db8-3ed75acd5c65/volumes" Oct 30 14:52:03.640237 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:52:03.640173 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:52:32.826666 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:32.826635 5824 scope.go:122] "RemoveContainer" containerID="35b8ad2dbbe45d29c835ef95051b460783d0a414279db0752f7a22fa3dea8f6c" Oct 30 14:52:32.837069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:32.837050 5824 scope.go:122] "RemoveContainer" containerID="0aabf738ec426a46e9edd0af8bf8cb4864b28e042ea033139c0198051438d028" Oct 30 14:52:32.846026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:32.846004 5824 scope.go:122] "RemoveContainer" containerID="10928c60e68a97155493e4579c5c58484064cc2f7795ec620ec9856199ef8ad6" Oct 30 14:52:32.853918 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:32.853900 5824 scope.go:122] "RemoveContainer" containerID="d7133909b2db4b564983cfca1b4fe279e588ce86b39f5ec33e8ab11d0217ff4e" Oct 30 14:52:32.872054 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:32.871327 5824 scope.go:122] "RemoveContainer" containerID="3bf596b4fd6bd1a0cd3b18b19a41578ea2cd40bed85897f68f2b1d772fb18b21" Oct 30 14:52:33.322292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.322260 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw"] Oct 30 14:52:33.322628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.322617 5824 cpu_manager.go:414] "containerMap: removing container" podUID="33b982f2-9323-4aec-83a5-63ecaf23463c" containerName="revoke-keys" Oct 30 14:52:33.322674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.322629 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b982f2-9323-4aec-83a5-63ecaf23463c" containerName="revoke-keys" Oct 30 14:52:33.322729 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.322720 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b982f2-9323-4aec-83a5-63ecaf23463c" containerName="revoke-keys" Oct 30 14:52:33.327398 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.327383 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.338889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.338866 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.338986 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.338908 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2wq\" (UniqueName: \"kubernetes.io/projected/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-kube-api-access-5t2wq\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.339425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.339391 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.344761 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.344737 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw"] Oct 30 14:52:33.356822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.356801 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:52:33.356822 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.356801 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:52:33.356952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.356837 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:52:33.357778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.357764 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:52:33.440529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.440509 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.440616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.440538 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2wq\" (UniqueName: \"kubernetes.io/projected/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-kube-api-access-5t2wq\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.440616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.440584 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.441082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.441066 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.442689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.442674 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.466017 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.465997 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2wq\" (UniqueName: \"kubernetes.io/projected/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-kube-api-access-5t2wq\") pod \"maas-api-revoke-sub-c1e271a9-lztnw\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.639503 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.639437 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:33.639503 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:52:33.639442 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:52:33.935158 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:52:33.935127 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48834e4f_f5f0_43eb_b2ad_97e50fce68a5.slice/crio-3211c990bf21984e49a382f605c62d2c00a49c5e03575f04c3cd1e0245500ea1 WatchSource:0}: Error finding container 3211c990bf21984e49a382f605c62d2c00a49c5e03575f04c3cd1e0245500ea1: Status 404 returned error can't find the container with id 3211c990bf21984e49a382f605c62d2c00a49c5e03575f04c3cd1e0245500ea1 Oct 30 14:52:33.935577 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:33.935323 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw"] Oct 30 14:52:34.294372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:34.294313 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" event={"ID":"48834e4f-f5f0-43eb-b2ad-97e50fce68a5","Type":"ContainerStarted","Data":"27c17a43cb0f2067f6156d443feab211084082ca098860b65aebd8839cd5da0b"} Oct 30 14:52:34.294372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:34.294370 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" event={"ID":"48834e4f-f5f0-43eb-b2ad-97e50fce68a5","Type":"ContainerStarted","Data":"3211c990bf21984e49a382f605c62d2c00a49c5e03575f04c3cd1e0245500ea1"} Oct 30 14:52:34.346126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:34.346078 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" podStartSLOduration=1.34606588 podStartE2EDuration="1.34606588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:34.343073241 +0000 UTC m=+3784.143578710" watchObservedRunningTime="2026-10-30 14:52:34.34606588 +0000 UTC m=+3784.146571373" Oct 30 14:52:35.305963 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:35.305915 5824 generic.go:356] "Generic (PLEG): container finished" podID="48834e4f-f5f0-43eb-b2ad-97e50fce68a5" containerID="27c17a43cb0f2067f6156d443feab211084082ca098860b65aebd8839cd5da0b" exitCode=0 Oct 30 14:52:35.306484 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:35.306140 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" event={"ID":"48834e4f-f5f0-43eb-b2ad-97e50fce68a5","Type":"ContainerDied","Data":"27c17a43cb0f2067f6156d443feab211084082ca098860b65aebd8839cd5da0b"} Oct 30 14:52:36.443500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.443475 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:36.465615 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.465590 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-kube-api-access-5t2wq\" (UniqueName: \"kubernetes.io/projected/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-kube-api-access-5t2wq\") pod \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " Oct 30 14:52:36.465764 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.465746 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-openshift-ca\") pod \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " Oct 30 14:52:36.465826 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.465793 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-xks-ca\") pod \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\" (UID: \"48834e4f-f5f0-43eb-b2ad-97e50fce68a5\") " Oct 30 14:52:36.466096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.466074 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-openshift-ca" pod "48834e4f-f5f0-43eb-b2ad-97e50fce68a5" (UID: "48834e4f-f5f0-43eb-b2ad-97e50fce68a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:52:36.467708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.467686 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-kube-api-access-5t2wq" pod "48834e4f-f5f0-43eb-b2ad-97e50fce68a5" (UID: "48834e4f-f5f0-43eb-b2ad-97e50fce68a5"). InnerVolumeSpecName "kube-api-access-5t2wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:52:36.467843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.467827 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-xks-ca" pod "48834e4f-f5f0-43eb-b2ad-97e50fce68a5" (UID: "48834e4f-f5f0-43eb-b2ad-97e50fce68a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:52:36.566769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.566709 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:52:36.566769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.566734 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:52:36.566769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:36.566742 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2wq\" (UniqueName: \"kubernetes.io/projected/48834e4f-f5f0-43eb-b2ad-97e50fce68a5-kube-api-access-5t2wq\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:52:37.319268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:37.319232 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" Oct 30 14:52:37.319268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:37.319248 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw" event={"ID":"48834e4f-f5f0-43eb-b2ad-97e50fce68a5","Type":"ContainerDied","Data":"3211c990bf21984e49a382f605c62d2c00a49c5e03575f04c3cd1e0245500ea1"} Oct 30 14:52:37.319483 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:37.319300 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3211c990bf21984e49a382f605c62d2c00a49c5e03575f04c3cd1e0245500ea1" Oct 30 14:52:41.705707 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.705634 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x"] Oct 30 14:52:41.706081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.706003 5824 cpu_manager.go:414] "containerMap: removing container" podUID="48834e4f-f5f0-43eb-b2ad-97e50fce68a5" containerName="revoke-keys" Oct 30 14:52:41.706081 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.706014 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48834e4f-f5f0-43eb-b2ad-97e50fce68a5" containerName="revoke-keys" Oct 30 14:52:41.707843 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.706663 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="48834e4f-f5f0-43eb-b2ad-97e50fce68a5" containerName="revoke-keys" Oct 30 14:52:41.711509 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.711464 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.733063 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.733032 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x"] Oct 30 14:52:41.811946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.811916 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghbq\" (UniqueName: \"kubernetes.io/projected/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-kube-api-access-fghbq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.812089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.811969 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.812089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.812015 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.812089 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.812047 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.812275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.812094 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.812275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.812121 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.812275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.812156 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.812378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.812290 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.812378 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.812317 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.912902 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.912872 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.912923 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.912952 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.912976 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913026 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913012 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghbq\" (UniqueName: \"kubernetes.io/projected/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-kube-api-access-fghbq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913059 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913085 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913108 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913228 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913137 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913457 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913440 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913482 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913647 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913624 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913645 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.913708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.913690 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.915386 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.915359 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.915495 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.915451 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.938928 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.938905 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:41.939127 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:41.939097 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghbq\" (UniqueName: \"kubernetes.io/projected/2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca-kube-api-access-fghbq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d7k5x\" (UID: \"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:42.028719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:42.028669 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:42.341794 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:42.341772 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x"] Oct 30 14:52:42.344627 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:52:42.344601 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbddc3f_7abd_4aa5_a42b_f018fe7e65ca.slice/crio-0c9f2c67b9d2cbbf9a00a0b73c3e064bd6234ba70562218bde70cbdb223b8882 WatchSource:0}: Error finding container 0c9f2c67b9d2cbbf9a00a0b73c3e064bd6234ba70562218bde70cbdb223b8882: Status 404 returned error can't find the container with id 0c9f2c67b9d2cbbf9a00a0b73c3e064bd6234ba70562218bde70cbdb223b8882 Oct 30 14:52:42.346645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:42.346606 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:52:42.346748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:42.346689 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:52:42.346748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:42.346723 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:52:43.371223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:43.371164 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" event={"ID":"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca","Type":"ContainerStarted","Data":"6cf649fe883f8ce04b44d2b6eccec4c9995c668a9d3ed3196866be69f2940596"} Oct 30 14:52:43.371648 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:43.371244 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" event={"ID":"2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca","Type":"ContainerStarted","Data":"0c9f2c67b9d2cbbf9a00a0b73c3e064bd6234ba70562218bde70cbdb223b8882"} Oct 30 14:52:43.434121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:43.434070 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podStartSLOduration=2.4340564000000002 podStartE2EDuration="2.4340564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:43.432838279 +0000 UTC m=+3793.233343735" watchObservedRunningTime="2026-10-30 14:52:43.4340564 +0000 UTC m=+3793.234561843" Oct 30 14:52:44.029510 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:44.029476 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:44.030603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:44.030574 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 14:52:44.030735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:44.030620 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 14:52:45.029381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:45.029347 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 14:52:45.029757 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:45.029397 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 14:52:46.029774 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:46.029731 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 14:52:46.030177 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:46.029800 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 14:52:47.029402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:47.029366 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 14:52:47.029572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:47.029419 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 14:52:48.029961 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:48.029923 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 14:52:48.030464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:48.029980 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 14:52:49.029313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:49.029263 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 14:52:49.029532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:49.029336 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 14:52:51.014328 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:51.014287 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:51.014838 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:51.014367 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cl2qx" podUID="6bc4102f-81bc-4831-bf65-fcd79c204148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:51.029957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:51.029927 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:51.030096 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:51.029993 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:51.144353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:51.144313 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:51.144626 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:51.144581 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gqq8b" podUID="d5057b65-74d7-4c84-b500-ba697b16459f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:52.071340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:52.071303 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:52.071833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:52.071384 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:53.071786 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:53.071751 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:53.072321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:53.071809 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:54.112337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:54.112297 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:52:54.112815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:54.112354 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:52:54.645570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:54.645541 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:54.645904 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:54.645884 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:54.646588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:54.646567 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" Oct 30 14:52:56.627044 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.627009 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns"] Oct 30 14:52:56.631084 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.631059 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.649621 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.649591 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns"] Oct 30 14:52:56.660766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.660729 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:52:56.660910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.660766 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:52:56.660910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.660741 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:52:56.663408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.663391 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:52:56.789382 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.789348 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.789535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.789455 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbt77\" (UniqueName: \"kubernetes.io/projected/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-kube-api-access-cbt77\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.789535 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.789504 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.890801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.890728 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.890935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.890880 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbt77\" (UniqueName: \"kubernetes.io/projected/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-kube-api-access-cbt77\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.890935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.890926 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.891497 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.891476 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.892942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.892924 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.917040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.917013 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbt77\" (UniqueName: \"kubernetes.io/projected/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-kube-api-access-cbt77\") pod \"maas-api-revoke-sub-2e1f5b5f-v24ns\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:56.944441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:56.944421 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:57.242908 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:57.242880 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns"] Oct 30 14:52:57.243573 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:52:57.243551 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ccd6ed_8d41_433a_94ce_80251c3d3be6.slice/crio-b5207100db714f5a80946bf1ab3f693d1cb5d0e3efe7c111f152c44ee35a8d16 WatchSource:0}: Error finding container b5207100db714f5a80946bf1ab3f693d1cb5d0e3efe7c111f152c44ee35a8d16: Status 404 returned error can't find the container with id b5207100db714f5a80946bf1ab3f693d1cb5d0e3efe7c111f152c44ee35a8d16 Oct 30 14:52:57.455069 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:57.455037 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" event={"ID":"d0ccd6ed-8d41-433a-94ce-80251c3d3be6","Type":"ContainerStarted","Data":"b5207100db714f5a80946bf1ab3f693d1cb5d0e3efe7c111f152c44ee35a8d16"} Oct 30 14:52:58.461538 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:58.461502 5824 generic.go:356] "Generic (PLEG): container finished" podID="d0ccd6ed-8d41-433a-94ce-80251c3d3be6" containerID="5faed051d9283f355c7b8562284f20f47000977720828344e80c6d7268e17a0d" exitCode=0 Oct 30 14:52:58.461878 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:58.461585 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" event={"ID":"d0ccd6ed-8d41-433a-94ce-80251c3d3be6","Type":"ContainerDied","Data":"5faed051d9283f355c7b8562284f20f47000977720828344e80c6d7268e17a0d"} Oct 30 14:52:59.595444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.595417 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:52:59.716006 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.715982 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-openshift-ca\") pod \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " Oct 30 14:52:59.716162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.716079 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-kube-api-access-cbt77\" (UniqueName: \"kubernetes.io/projected/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-kube-api-access-cbt77\") pod \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " Oct 30 14:52:59.716162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.716108 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-xks-ca\") pod \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\" (UID: \"d0ccd6ed-8d41-433a-94ce-80251c3d3be6\") " Oct 30 14:52:59.716363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.716342 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-openshift-ca" pod "d0ccd6ed-8d41-433a-94ce-80251c3d3be6" (UID: "d0ccd6ed-8d41-433a-94ce-80251c3d3be6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:52:59.716458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.716440 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:52:59.718001 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.717947 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-kube-api-access-cbt77" pod "d0ccd6ed-8d41-433a-94ce-80251c3d3be6" (UID: "d0ccd6ed-8d41-433a-94ce-80251c3d3be6"). InnerVolumeSpecName "kube-api-access-cbt77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:52:59.718120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.718015 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-xks-ca" pod "d0ccd6ed-8d41-433a-94ce-80251c3d3be6" (UID: "d0ccd6ed-8d41-433a-94ce-80251c3d3be6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:52:59.817301 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.817281 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbt77\" (UniqueName: \"kubernetes.io/projected/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-kube-api-access-cbt77\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:52:59.817396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:52:59.817304 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ccd6ed-8d41-433a-94ce-80251c3d3be6-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:00.473693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:00.473671 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" Oct 30 14:53:00.473816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:00.473716 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns" event={"ID":"d0ccd6ed-8d41-433a-94ce-80251c3d3be6","Type":"ContainerDied","Data":"b5207100db714f5a80946bf1ab3f693d1cb5d0e3efe7c111f152c44ee35a8d16"} Oct 30 14:53:00.473816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:00.473787 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5207100db714f5a80946bf1ab3f693d1cb5d0e3efe7c111f152c44ee35a8d16" Oct 30 14:53:03.641773 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:53:03.641740 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:53:07.838463 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.838427 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj"] Oct 30 14:53:07.838863 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.838850 5824 cpu_manager.go:414] "containerMap: removing container" podUID="d0ccd6ed-8d41-433a-94ce-80251c3d3be6" containerName="revoke-keys" Oct 30 14:53:07.838907 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.838864 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ccd6ed-8d41-433a-94ce-80251c3d3be6" containerName="revoke-keys" Oct 30 14:53:07.838960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.838951 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ccd6ed-8d41-433a-94ce-80251c3d3be6" containerName="revoke-keys" Oct 30 14:53:07.842157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.842130 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:07.861399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.861372 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj"] Oct 30 14:53:07.872227 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.872172 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:53:07.872400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.872222 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:53:07.872492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.872279 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:53:07.872492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.872284 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:53:07.983310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.983280 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6x4r\" (UniqueName: \"kubernetes.io/projected/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-kube-api-access-x6x4r\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:07.983310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.983312 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:07.983496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:07.983406 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:08.084816 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.084780 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:08.084981 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.084956 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6x4r\" (UniqueName: \"kubernetes.io/projected/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-kube-api-access-x6x4r\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:08.085040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.085007 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:08.085460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.085436 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:08.087515 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.087492 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:08.110903 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.110852 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6x4r\" (UniqueName: \"kubernetes.io/projected/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-kube-api-access-x6x4r\") pod \"maas-api-revoke-sub-ec843018-79hfj\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:08.155365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.155341 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:08.455606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.455576 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj"] Oct 30 14:53:08.456276 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:53:08.456250 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8bf930_d23a_4d0e_9bc2_fbebcd4fa9bc.slice/crio-d0c8ded3147688a7ea7d8b5c8512a3e661b5bb1a27486bec0c7c9fdc0b527b60 WatchSource:0}: Error finding container d0c8ded3147688a7ea7d8b5c8512a3e661b5bb1a27486bec0c7c9fdc0b527b60: Status 404 returned error can't find the container with id d0c8ded3147688a7ea7d8b5c8512a3e661b5bb1a27486bec0c7c9fdc0b527b60 Oct 30 14:53:08.519088 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:08.519059 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" event={"ID":"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc","Type":"ContainerStarted","Data":"d0c8ded3147688a7ea7d8b5c8512a3e661b5bb1a27486bec0c7c9fdc0b527b60"} Oct 30 14:53:09.528809 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:09.528723 5824 generic.go:356] "Generic (PLEG): container finished" podID="ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc" containerID="6ecfc231dc57ac39d39d2d5fe263962416c71b59c4c6e20b6c3c2b87529646d0" exitCode=0 Oct 30 14:53:09.529176 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:09.528808 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" event={"ID":"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc","Type":"ContainerDied","Data":"6ecfc231dc57ac39d39d2d5fe263962416c71b59c4c6e20b6c3c2b87529646d0"} Oct 30 14:53:10.680040 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.680015 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:10.813646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.813557 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-openshift-ca\") pod \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " Oct 30 14:53:10.813811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.813652 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-kube-api-access-x6x4r\" (UniqueName: \"kubernetes.io/projected/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-kube-api-access-x6x4r\") pod \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " Oct 30 14:53:10.813811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.813729 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-xks-ca\") pod \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\" (UID: \"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc\") " Oct 30 14:53:10.814011 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.813984 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-openshift-ca" pod "ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc" (UID: "ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:53:10.815929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.815897 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-kube-api-access-x6x4r" pod "ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc" (UID: "ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc"). InnerVolumeSpecName "kube-api-access-x6x4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:53:10.816219 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.816175 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-xks-ca" pod "ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc" (UID: "ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:53:10.914811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.914781 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:10.914811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.914809 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:10.914959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:10.914826 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6x4r\" (UniqueName: \"kubernetes.io/projected/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc-kube-api-access-x6x4r\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:11.549337 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:11.549297 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" event={"ID":"ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc","Type":"ContainerDied","Data":"d0c8ded3147688a7ea7d8b5c8512a3e661b5bb1a27486bec0c7c9fdc0b527b60"} Oct 30 14:53:11.549628 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:11.549608 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c8ded3147688a7ea7d8b5c8512a3e661b5bb1a27486bec0c7c9fdc0b527b60" Oct 30 14:53:11.549830 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:11.549815 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj" Oct 30 14:53:32.915419 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:32.915385 5824 scope.go:122] "RemoveContainer" containerID="b59400a8214552b1747c1a21ae930441ec8279b2e0f62b7dec365223dc6050a0" Oct 30 14:53:32.923727 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:32.923703 5824 scope.go:122] "RemoveContainer" containerID="ead6d75c066ccc671b43ac586586153db0295d4defdf05e090bc7cb95154d0f0" Oct 30 14:53:32.931603 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:32.931586 5824 scope.go:122] "RemoveContainer" containerID="a5304bd69f25b67bd17cee45f1dab8de0ed1a843d28203efe424ef868b843db7" Oct 30 14:53:32.939413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:32.939392 5824 scope.go:122] "RemoveContainer" containerID="9fc509896e8f6e3778aa4400c07fa96b098655fc0e9a5b550e31fbf0448bdf30" Oct 30 14:53:32.946703 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:32.946683 5824 scope.go:122] "RemoveContainer" containerID="45ec94fcc9ee51f27239b6470ffc451a0d3067c91d5a2a62e12e3c68ffc778c3" Oct 30 14:53:32.954015 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:32.953992 5824 scope.go:122] "RemoveContainer" containerID="f90c9a8e7bee8b26433151e6fdf4b4f4521873a553c12fbb5ff347c04881c297" Oct 30 14:53:32.961175 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:32.961157 5824 scope.go:122] "RemoveContainer" containerID="f010fed4ecd0250c9496df2b3601c384e45ab3fc1b623f5e2013bd307bb41fd1" Oct 30 14:53:32.968814 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:32.968797 5824 scope.go:122] "RemoveContainer" containerID="df41910aba43f3c780ddd774197f93904291cf22b6e36b605b68ed48ff487dc4" Oct 30 14:53:33.639325 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:53:33.639293 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:53:41.050410 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:41.050377 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9"] Oct 30 14:53:41.063249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:41.063217 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f9e65e3-7lpl9"] Oct 30 14:53:42.315331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:42.315290 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b1d0d4-4f9a-4d77-9656-9dcd3167c380" path="/var/lib/kubelet/pods/63b1d0d4-4f9a-4d77-9656-9dcd3167c380/volumes" Oct 30 14:53:44.662153 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.662123 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4"] Oct 30 14:53:44.662573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.662540 5824 cpu_manager.go:414] "containerMap: removing container" podUID="ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc" containerName="revoke-keys" Oct 30 14:53:44.662573 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.662553 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc" containerName="revoke-keys" Oct 30 14:53:44.662669 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.662632 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc" containerName="revoke-keys" Oct 30 14:53:44.666151 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.666131 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.684242 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.684213 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4"] Oct 30 14:53:44.695844 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.695821 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:53:44.695947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.695857 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:53:44.695947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.695914 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:53:44.696328 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.696312 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:53:44.814519 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.814487 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.814655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.814599 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.814655 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.814623 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftfgz\" (UniqueName: \"kubernetes.io/projected/fd4973a1-f089-49ed-b1db-8eef28bc25ad-kube-api-access-ftfgz\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.915281 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.915186 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.915436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.915304 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.915436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.915335 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftfgz\" (UniqueName: \"kubernetes.io/projected/fd4973a1-f089-49ed-b1db-8eef28bc25ad-kube-api-access-ftfgz\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.915933 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.915913 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.917524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.917505 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.941990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.941967 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftfgz\" (UniqueName: \"kubernetes.io/projected/fd4973a1-f089-49ed-b1db-8eef28bc25ad-kube-api-access-ftfgz\") pod \"maas-api-revoke-sub-046e1343-hq2t4\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:44.978965 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:44.978942 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:45.289701 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:45.289659 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4"] Oct 30 14:53:45.293312 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:53:45.293285 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4973a1_f089_49ed_b1db_8eef28bc25ad.slice/crio-bbecd4fd9bb1a6dd6949bbeae1237c31240bf71314646569c8a006feb1adc3ff WatchSource:0}: Error finding container bbecd4fd9bb1a6dd6949bbeae1237c31240bf71314646569c8a006feb1adc3ff: Status 404 returned error can't find the container with id bbecd4fd9bb1a6dd6949bbeae1237c31240bf71314646569c8a006feb1adc3ff Oct 30 14:53:45.771864 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:45.771834 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" event={"ID":"fd4973a1-f089-49ed-b1db-8eef28bc25ad","Type":"ContainerStarted","Data":"618a8cded7990162b3b0af5fbd594934bad29919f316b10c860fdf9234ffea4f"} Oct 30 14:53:45.772265 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:45.771895 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" event={"ID":"fd4973a1-f089-49ed-b1db-8eef28bc25ad","Type":"ContainerStarted","Data":"bbecd4fd9bb1a6dd6949bbeae1237c31240bf71314646569c8a006feb1adc3ff"} Oct 30 14:53:45.818636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:45.818595 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" podStartSLOduration=1.818582536 podStartE2EDuration="1.818582536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:53:45.817296095 +0000 UTC m=+3855.617801537" watchObservedRunningTime="2026-10-30 14:53:45.818582536 +0000 UTC m=+3855.619087979" Oct 30 14:53:46.778478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:46.778448 5824 generic.go:356] "Generic (PLEG): container finished" podID="fd4973a1-f089-49ed-b1db-8eef28bc25ad" containerID="618a8cded7990162b3b0af5fbd594934bad29919f316b10c860fdf9234ffea4f" exitCode=0 Oct 30 14:53:46.778942 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:46.778531 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" event={"ID":"fd4973a1-f089-49ed-b1db-8eef28bc25ad","Type":"ContainerDied","Data":"618a8cded7990162b3b0af5fbd594934bad29919f316b10c860fdf9234ffea4f"} Oct 30 14:53:47.916743 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:47.916718 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:48.045157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.045085 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-openshift-ca\") pod \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " Oct 30 14:53:48.045157 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.045139 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4973a1-f089-49ed-b1db-8eef28bc25ad-kube-api-access-ftfgz\" (UniqueName: \"kubernetes.io/projected/fd4973a1-f089-49ed-b1db-8eef28bc25ad-kube-api-access-ftfgz\") pod \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " Oct 30 14:53:48.045413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.045231 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-xks-ca\") pod \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\" (UID: \"fd4973a1-f089-49ed-b1db-8eef28bc25ad\") " Oct 30 14:53:48.045543 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.045519 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-openshift-ca" pod "fd4973a1-f089-49ed-b1db-8eef28bc25ad" (UID: "fd4973a1-f089-49ed-b1db-8eef28bc25ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:53:48.047184 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.047160 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-xks-ca" pod "fd4973a1-f089-49ed-b1db-8eef28bc25ad" (UID: "fd4973a1-f089-49ed-b1db-8eef28bc25ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:53:48.047283 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.047226 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4973a1-f089-49ed-b1db-8eef28bc25ad-kube-api-access-ftfgz" pod "fd4973a1-f089-49ed-b1db-8eef28bc25ad" (UID: "fd4973a1-f089-49ed-b1db-8eef28bc25ad"). InnerVolumeSpecName "kube-api-access-ftfgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:53:48.146832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.146804 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:48.146832 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.146830 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftfgz\" (UniqueName: \"kubernetes.io/projected/fd4973a1-f089-49ed-b1db-8eef28bc25ad-kube-api-access-ftfgz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:48.146984 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.146841 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd4973a1-f089-49ed-b1db-8eef28bc25ad-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:48.790785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.790748 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" event={"ID":"fd4973a1-f089-49ed-b1db-8eef28bc25ad","Type":"ContainerDied","Data":"bbecd4fd9bb1a6dd6949bbeae1237c31240bf71314646569c8a006feb1adc3ff"} Oct 30 14:53:48.790947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.790800 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbecd4fd9bb1a6dd6949bbeae1237c31240bf71314646569c8a006feb1adc3ff" Oct 30 14:53:48.790947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:48.790760 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4" Oct 30 14:53:54.997068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:54.997025 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn"] Oct 30 14:53:54.997659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:54.997515 5824 cpu_manager.go:414] "containerMap: removing container" podUID="fd4973a1-f089-49ed-b1db-8eef28bc25ad" containerName="revoke-keys" Oct 30 14:53:54.997659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:54.997530 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4973a1-f089-49ed-b1db-8eef28bc25ad" containerName="revoke-keys" Oct 30 14:53:54.997659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:54.997591 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4973a1-f089-49ed-b1db-8eef28bc25ad" containerName="revoke-keys" Oct 30 14:53:55.000623 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.000601 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.019209 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.019163 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn"] Oct 30 14:53:55.030613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.030587 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:53:55.030613 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.030591 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:53:55.030833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.030595 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:53:55.030833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.030692 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:53:55.103134 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.103092 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.103299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.103146 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.103348 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.103322 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d2rt\" (UniqueName: \"kubernetes.io/projected/beb8f498-7601-4fa2-99e9-48ac5faacbcb-kube-api-access-7d2rt\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.204365 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.204336 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.204492 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.204382 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.204588 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.204566 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d2rt\" (UniqueName: \"kubernetes.io/projected/beb8f498-7601-4fa2-99e9-48ac5faacbcb-kube-api-access-7d2rt\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.205068 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.205050 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.206618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.206590 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.230441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.230414 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d2rt\" (UniqueName: \"kubernetes.io/projected/beb8f498-7601-4fa2-99e9-48ac5faacbcb-kube-api-access-7d2rt\") pod \"maas-api-revoke-sub-d3fba056-2vbjn\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.313609 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.313556 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:55.610295 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:53:55.610258 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb8f498_7601_4fa2_99e9_48ac5faacbcb.slice/crio-93e207d5431479c545165f29b9e83af334eb78482ce064da7d9e35847b821814 WatchSource:0}: Error finding container 93e207d5431479c545165f29b9e83af334eb78482ce064da7d9e35847b821814: Status 404 returned error can't find the container with id 93e207d5431479c545165f29b9e83af334eb78482ce064da7d9e35847b821814 Oct 30 14:53:55.610408 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.610350 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn"] Oct 30 14:53:55.828109 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:55.828076 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" event={"ID":"beb8f498-7601-4fa2-99e9-48ac5faacbcb","Type":"ContainerStarted","Data":"93e207d5431479c545165f29b9e83af334eb78482ce064da7d9e35847b821814"} Oct 30 14:53:56.837234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:56.837120 5824 generic.go:356] "Generic (PLEG): container finished" podID="beb8f498-7601-4fa2-99e9-48ac5faacbcb" containerID="fe907441b7cf05639dce956e30397bb0a1683e52106f0f50ed36fcacdc4c2839" exitCode=0 Oct 30 14:53:56.837234 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:56.837208 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" event={"ID":"beb8f498-7601-4fa2-99e9-48ac5faacbcb","Type":"ContainerDied","Data":"fe907441b7cf05639dce956e30397bb0a1683e52106f0f50ed36fcacdc4c2839"} Oct 30 14:53:58.259363 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.259336 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:58.334379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.334348 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb8f498-7601-4fa2-99e9-48ac5faacbcb-kube-api-access-7d2rt\" (UniqueName: \"kubernetes.io/projected/beb8f498-7601-4fa2-99e9-48ac5faacbcb-kube-api-access-7d2rt\") pod \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " Oct 30 14:53:58.334540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.334415 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-xks-ca\") pod \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " Oct 30 14:53:58.334540 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.334489 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-openshift-ca\") pod \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\" (UID: \"beb8f498-7601-4fa2-99e9-48ac5faacbcb\") " Oct 30 14:53:58.335000 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.334968 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-openshift-ca" pod "beb8f498-7601-4fa2-99e9-48ac5faacbcb" (UID: "beb8f498-7601-4fa2-99e9-48ac5faacbcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:53:58.336346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.336328 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-xks-ca" pod "beb8f498-7601-4fa2-99e9-48ac5faacbcb" (UID: "beb8f498-7601-4fa2-99e9-48ac5faacbcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:53:58.336821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.336801 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb8f498-7601-4fa2-99e9-48ac5faacbcb-kube-api-access-7d2rt" pod "beb8f498-7601-4fa2-99e9-48ac5faacbcb" (UID: "beb8f498-7601-4fa2-99e9-48ac5faacbcb"). InnerVolumeSpecName "kube-api-access-7d2rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:53:58.435993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.435946 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d2rt\" (UniqueName: \"kubernetes.io/projected/beb8f498-7601-4fa2-99e9-48ac5faacbcb-kube-api-access-7d2rt\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:58.435993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.435967 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:58.435993 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.435977 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8f498-7601-4fa2-99e9-48ac5faacbcb-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:53:58.854920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.854881 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" Oct 30 14:53:58.854920 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.854913 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn" event={"ID":"beb8f498-7601-4fa2-99e9-48ac5faacbcb","Type":"ContainerDied","Data":"93e207d5431479c545165f29b9e83af334eb78482ce064da7d9e35847b821814"} Oct 30 14:53:58.855141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:53:58.854960 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93e207d5431479c545165f29b9e83af334eb78482ce064da7d9e35847b821814" Oct 30 14:54:03.639283 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:54:03.639238 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:54:26.766206 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.766111 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp"] Oct 30 14:54:26.766881 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.766854 5824 cpu_manager.go:414] "containerMap: removing container" podUID="beb8f498-7601-4fa2-99e9-48ac5faacbcb" containerName="revoke-keys" Oct 30 14:54:26.766927 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.766886 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb8f498-7601-4fa2-99e9-48ac5faacbcb" containerName="revoke-keys" Oct 30 14:54:26.767083 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.767068 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb8f498-7601-4fa2-99e9-48ac5faacbcb" containerName="revoke-keys" Oct 30 14:54:26.771047 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.771025 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:26.789009 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.788982 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp"] Oct 30 14:54:26.798529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.798493 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:54:26.798529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.798493 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:54:26.798686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.798597 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:54:26.803924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.803889 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:54:26.919579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.919546 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:26.919724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.919589 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/c4f61cb3-3582-40a0-8384-1ecd709a115f-kube-api-access-gsdhs\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:26.919724 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:26.919663 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:27.020522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:27.020450 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:27.020522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:27.020501 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/c4f61cb3-3582-40a0-8384-1ecd709a115f-kube-api-access-gsdhs\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:27.020717 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:27.020550 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:27.021145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:27.021124 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:27.022612 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:27.022590 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:27.046100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:27.046081 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/c4f61cb3-3582-40a0-8384-1ecd709a115f-kube-api-access-gsdhs\") pod \"maas-api-revoke-sub-ae594979-8h8dp\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:27.084136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:27.084112 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:27.381619 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:27.381594 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp"] Oct 30 14:54:27.382169 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:54:27.382141 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f61cb3_3582_40a0_8384_1ecd709a115f.slice/crio-86550d94d4fcb82e9a5ecb73e7a48ae566ec7f3ad8cbfc4e9b6cf0d2273726d0 WatchSource:0}: Error finding container 86550d94d4fcb82e9a5ecb73e7a48ae566ec7f3ad8cbfc4e9b6cf0d2273726d0: Status 404 returned error can't find the container with id 86550d94d4fcb82e9a5ecb73e7a48ae566ec7f3ad8cbfc4e9b6cf0d2273726d0 Oct 30 14:54:28.007529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:28.007494 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" event={"ID":"c4f61cb3-3582-40a0-8384-1ecd709a115f","Type":"ContainerStarted","Data":"48c1e667eacc61b55ac7dc0370d23fac2106be1ec81806f0f02c02e45a8e76bf"} Oct 30 14:54:28.007875 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:28.007559 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" event={"ID":"c4f61cb3-3582-40a0-8384-1ecd709a115f","Type":"ContainerStarted","Data":"86550d94d4fcb82e9a5ecb73e7a48ae566ec7f3ad8cbfc4e9b6cf0d2273726d0"} Oct 30 14:54:28.052344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:28.052304 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" podStartSLOduration=2.052288316 podStartE2EDuration="2.052288316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:28.051410981 +0000 UTC m=+3897.851916424" watchObservedRunningTime="2026-10-30 14:54:28.052288316 +0000 UTC m=+3897.852793754" Oct 30 14:54:29.014525 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:29.014491 5824 generic.go:356] "Generic (PLEG): container finished" podID="c4f61cb3-3582-40a0-8384-1ecd709a115f" containerID="48c1e667eacc61b55ac7dc0370d23fac2106be1ec81806f0f02c02e45a8e76bf" exitCode=0 Oct 30 14:54:29.014966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:29.014545 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" event={"ID":"c4f61cb3-3582-40a0-8384-1ecd709a115f","Type":"ContainerDied","Data":"48c1e667eacc61b55ac7dc0370d23fac2106be1ec81806f0f02c02e45a8e76bf"} Oct 30 14:54:30.148823 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.148796 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:30.249049 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.249022 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-xks-ca\") pod \"c4f61cb3-3582-40a0-8384-1ecd709a115f\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " Oct 30 14:54:30.249181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.249066 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f61cb3-3582-40a0-8384-1ecd709a115f-kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/c4f61cb3-3582-40a0-8384-1ecd709a115f-kube-api-access-gsdhs\") pod \"c4f61cb3-3582-40a0-8384-1ecd709a115f\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " Oct 30 14:54:30.249181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.249097 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-openshift-ca\") pod \"c4f61cb3-3582-40a0-8384-1ecd709a115f\" (UID: \"c4f61cb3-3582-40a0-8384-1ecd709a115f\") " Oct 30 14:54:30.249507 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.249483 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-openshift-ca" pod "c4f61cb3-3582-40a0-8384-1ecd709a115f" (UID: "c4f61cb3-3582-40a0-8384-1ecd709a115f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:54:30.251070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.251045 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-xks-ca" pod "c4f61cb3-3582-40a0-8384-1ecd709a115f" (UID: "c4f61cb3-3582-40a0-8384-1ecd709a115f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:54:30.251133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.251052 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f61cb3-3582-40a0-8384-1ecd709a115f-kube-api-access-gsdhs" pod "c4f61cb3-3582-40a0-8384-1ecd709a115f" (UID: "c4f61cb3-3582-40a0-8384-1ecd709a115f"). InnerVolumeSpecName "kube-api-access-gsdhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:54:30.350272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.350251 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:54:30.350272 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.350271 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4f61cb3-3582-40a0-8384-1ecd709a115f-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:54:30.350400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:30.350281 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsdhs\" (UniqueName: \"kubernetes.io/projected/c4f61cb3-3582-40a0-8384-1ecd709a115f-kube-api-access-gsdhs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:54:31.026785 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:31.026763 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" Oct 30 14:54:31.026960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:31.026791 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp" event={"ID":"c4f61cb3-3582-40a0-8384-1ecd709a115f","Type":"ContainerDied","Data":"86550d94d4fcb82e9a5ecb73e7a48ae566ec7f3ad8cbfc4e9b6cf0d2273726d0"} Oct 30 14:54:31.026960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:31.026841 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86550d94d4fcb82e9a5ecb73e7a48ae566ec7f3ad8cbfc4e9b6cf0d2273726d0" Oct 30 14:54:33.014966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:33.014927 5824 scope.go:122] "RemoveContainer" containerID="e373122c6b253ce6d47c7d45f0d9418d4f73267e3886356986a7b43c6e5c3c28" Oct 30 14:54:33.024829 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:33.024805 5824 scope.go:122] "RemoveContainer" containerID="f892b7b72ca4db2d1216a36316bd6339c5e8b850ca4f160742fb14e13654f703" Oct 30 14:54:33.639624 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:54:33.639588 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:54:55.299570 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.299536 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg"] Oct 30 14:54:55.300167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.300146 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c4f61cb3-3582-40a0-8384-1ecd709a115f" containerName="revoke-keys" Oct 30 14:54:55.300293 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.300167 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f61cb3-3582-40a0-8384-1ecd709a115f" containerName="revoke-keys" Oct 30 14:54:55.300353 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.300307 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f61cb3-3582-40a0-8384-1ecd709a115f" containerName="revoke-keys" Oct 30 14:54:55.303498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.303477 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.321429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.321405 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg"] Oct 30 14:54:55.322693 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.322666 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:54:55.322792 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.322712 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:54:55.322924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.322900 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:54:55.332771 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.332747 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:54:55.359396 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.359373 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.359545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.359415 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79dm\" (UniqueName: \"kubernetes.io/projected/75f817cf-e772-44d7-a474-3a2ffc8c4368-kube-api-access-h79dm\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.359545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.359459 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.460625 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.460597 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79dm\" (UniqueName: \"kubernetes.io/projected/75f817cf-e772-44d7-a474-3a2ffc8c4368-kube-api-access-h79dm\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.460763 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.460634 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.460763 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.460702 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.461299 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.461279 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.462890 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.462869 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.486946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.486923 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79dm\" (UniqueName: \"kubernetes.io/projected/75f817cf-e772-44d7-a474-3a2ffc8c4368-kube-api-access-h79dm\") pod \"maas-api-revoke-sub-a4aef7d7-46wmg\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.615905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.615856 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:55.914616 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:55.914580 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg"] Oct 30 14:54:55.915019 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:54:55.914995 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f817cf_e772_44d7_a474_3a2ffc8c4368.slice/crio-04bd99c4da3954392516433cc4591b2aa5793911b925174f6689063e83174a60 WatchSource:0}: Error finding container 04bd99c4da3954392516433cc4591b2aa5793911b925174f6689063e83174a60: Status 404 returned error can't find the container with id 04bd99c4da3954392516433cc4591b2aa5793911b925174f6689063e83174a60 Oct 30 14:54:56.205579 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:56.205550 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" event={"ID":"75f817cf-e772-44d7-a474-3a2ffc8c4368","Type":"ContainerStarted","Data":"04bd99c4da3954392516433cc4591b2aa5793911b925174f6689063e83174a60"} Oct 30 14:54:57.213566 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:57.213533 5824 generic.go:356] "Generic (PLEG): container finished" podID="75f817cf-e772-44d7-a474-3a2ffc8c4368" containerID="a1c67d9728d8b7d108319affe916cf1a11a5faaf5d835f6b2e99c092011e6379" exitCode=0 Oct 30 14:54:57.213992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:57.213616 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" event={"ID":"75f817cf-e772-44d7-a474-3a2ffc8c4368","Type":"ContainerDied","Data":"a1c67d9728d8b7d108319affe916cf1a11a5faaf5d835f6b2e99c092011e6379"} Oct 30 14:54:58.354467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.354444 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:58.388288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.388264 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-xks-ca\") pod \"75f817cf-e772-44d7-a474-3a2ffc8c4368\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " Oct 30 14:54:58.388380 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.388362 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-openshift-ca\") pod \"75f817cf-e772-44d7-a474-3a2ffc8c4368\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " Oct 30 14:54:58.388458 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.388441 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f817cf-e772-44d7-a474-3a2ffc8c4368-kube-api-access-h79dm\" (UniqueName: \"kubernetes.io/projected/75f817cf-e772-44d7-a474-3a2ffc8c4368-kube-api-access-h79dm\") pod \"75f817cf-e772-44d7-a474-3a2ffc8c4368\" (UID: \"75f817cf-e772-44d7-a474-3a2ffc8c4368\") " Oct 30 14:54:58.388702 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.388675 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-openshift-ca" pod "75f817cf-e772-44d7-a474-3a2ffc8c4368" (UID: "75f817cf-e772-44d7-a474-3a2ffc8c4368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:54:58.388831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.388815 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:54:58.390250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.390225 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-xks-ca" pod "75f817cf-e772-44d7-a474-3a2ffc8c4368" (UID: "75f817cf-e772-44d7-a474-3a2ffc8c4368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:54:58.390356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.390323 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f817cf-e772-44d7-a474-3a2ffc8c4368-kube-api-access-h79dm" pod "75f817cf-e772-44d7-a474-3a2ffc8c4368" (UID: "75f817cf-e772-44d7-a474-3a2ffc8c4368"). InnerVolumeSpecName "kube-api-access-h79dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:54:58.489542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.489495 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79dm\" (UniqueName: \"kubernetes.io/projected/75f817cf-e772-44d7-a474-3a2ffc8c4368-kube-api-access-h79dm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:54:58.489542 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:58.489516 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f817cf-e772-44d7-a474-3a2ffc8c4368-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:54:59.226958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:59.226926 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" Oct 30 14:54:59.227158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:59.226958 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg" event={"ID":"75f817cf-e772-44d7-a474-3a2ffc8c4368","Type":"ContainerDied","Data":"04bd99c4da3954392516433cc4591b2aa5793911b925174f6689063e83174a60"} Oct 30 14:54:59.227158 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:54:59.227019 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04bd99c4da3954392516433cc4591b2aa5793911b925174f6689063e83174a60" Oct 30 14:55:02.053087 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:02.053051 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr"] Oct 30 14:55:02.066035 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:02.066006 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8ea5ce-9clsr"] Oct 30 14:55:02.314606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:02.314535 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81" path="/var/lib/kubelet/pods/2bf78b24-b4b5-4c3d-a8e0-8d669aab1c81/volumes" Oct 30 14:55:03.640533 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:55:03.640496 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:55:13.035557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:13.035518 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh"] Oct 30 14:55:13.048889 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:13.048849 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e287c30-rbcmh"] Oct 30 14:55:14.316145 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:14.316094 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc67a5f-2b3b-47a2-8729-0ef3fafd5259" path="/var/lib/kubelet/pods/bcc67a5f-2b3b-47a2-8729-0ef3fafd5259/volumes" Oct 30 14:55:25.506424 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.506392 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz"] Oct 30 14:55:25.506887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.506772 5824 cpu_manager.go:414] "containerMap: removing container" podUID="75f817cf-e772-44d7-a474-3a2ffc8c4368" containerName="revoke-keys" Oct 30 14:55:25.506887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.506783 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f817cf-e772-44d7-a474-3a2ffc8c4368" containerName="revoke-keys" Oct 30 14:55:25.506887 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.506872 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f817cf-e772-44d7-a474-3a2ffc8c4368" containerName="revoke-keys" Oct 30 14:55:25.510043 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.510021 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.532325 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.532296 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz"] Oct 30 14:55:25.542406 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.542385 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:55:25.542506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.542439 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:55:25.560154 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.560132 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:55:25.560263 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.560169 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:55:25.636809 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.636778 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47zj\" (UniqueName: \"kubernetes.io/projected/b20b4442-c4f3-4c68-8974-c497b27fc517-kube-api-access-r47zj\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.636917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.636841 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.636964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.636921 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.737336 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.737311 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47zj\" (UniqueName: \"kubernetes.io/projected/b20b4442-c4f3-4c68-8974-c497b27fc517-kube-api-access-r47zj\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.737449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.737356 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.737449 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.737381 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.738002 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.737979 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.739683 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.739662 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.763580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.763528 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47zj\" (UniqueName: \"kubernetes.io/projected/b20b4442-c4f3-4c68-8974-c497b27fc517-kube-api-access-r47zj\") pod \"maas-api-revoke-sub-79805ca9-ddssz\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:25.822524 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:25.822503 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:26.035478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.035403 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x"] Oct 30 14:55:26.043262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.043235 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm"] Oct 30 14:55:26.044807 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.044789 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e848ef7d-68q7x"] Oct 30 14:55:26.046404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.046381 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c002ceec-g7ckm"] Oct 30 14:55:26.120402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.120380 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz"] Oct 30 14:55:26.121058 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:55:26.121032 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20b4442_c4f3_4c68_8974_c497b27fc517.slice/crio-75c85ac1109d1e8ac0c5dd5909bbd90732e40ffbb0490dc55f40bd7e34ccb598 WatchSource:0}: Error finding container 75c85ac1109d1e8ac0c5dd5909bbd90732e40ffbb0490dc55f40bd7e34ccb598: Status 404 returned error can't find the container with id 75c85ac1109d1e8ac0c5dd5909bbd90732e40ffbb0490dc55f40bd7e34ccb598 Oct 30 14:55:26.315669 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.315612 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3584ff8d-b5ca-413c-8abb-52ad766ae183" path="/var/lib/kubelet/pods/3584ff8d-b5ca-413c-8abb-52ad766ae183/volumes" Oct 30 14:55:26.316017 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.316003 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af496e84-e61e-48cc-9a60-234c05473f92" path="/var/lib/kubelet/pods/af496e84-e61e-48cc-9a60-234c05473f92/volumes" Oct 30 14:55:26.398821 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.398791 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" event={"ID":"b20b4442-c4f3-4c68-8974-c497b27fc517","Type":"ContainerStarted","Data":"df47071374a8bf9326037d255b1072643a549ca92cbadec1ab2aa756c66a96b0"} Oct 30 14:55:26.398922 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.398849 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" event={"ID":"b20b4442-c4f3-4c68-8974-c497b27fc517","Type":"ContainerStarted","Data":"75c85ac1109d1e8ac0c5dd5909bbd90732e40ffbb0490dc55f40bd7e34ccb598"} Oct 30 14:55:26.443233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:26.443174 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" podStartSLOduration=1.4431585120000001 podStartE2EDuration="1.443158512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:26.441977326 +0000 UTC m=+3956.242482769" watchObservedRunningTime="2026-10-30 14:55:26.443158512 +0000 UTC m=+3956.243663957" Oct 30 14:55:27.406405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:27.406373 5824 generic.go:356] "Generic (PLEG): container finished" podID="b20b4442-c4f3-4c68-8974-c497b27fc517" containerID="df47071374a8bf9326037d255b1072643a549ca92cbadec1ab2aa756c66a96b0" exitCode=0 Oct 30 14:55:27.406828 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:27.406477 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" event={"ID":"b20b4442-c4f3-4c68-8974-c497b27fc517","Type":"ContainerDied","Data":"df47071374a8bf9326037d255b1072643a549ca92cbadec1ab2aa756c66a96b0"} Oct 30 14:55:28.543498 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.543474 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:28.661504 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.661472 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20b4442-c4f3-4c68-8974-c497b27fc517-kube-api-access-r47zj\" (UniqueName: \"kubernetes.io/projected/b20b4442-c4f3-4c68-8974-c497b27fc517-kube-api-access-r47zj\") pod \"b20b4442-c4f3-4c68-8974-c497b27fc517\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " Oct 30 14:55:28.661670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.661514 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-openshift-ca\") pod \"b20b4442-c4f3-4c68-8974-c497b27fc517\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " Oct 30 14:55:28.661670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.661597 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-xks-ca\") pod \"b20b4442-c4f3-4c68-8974-c497b27fc517\" (UID: \"b20b4442-c4f3-4c68-8974-c497b27fc517\") " Oct 30 14:55:28.661959 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.661931 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-openshift-ca" pod "b20b4442-c4f3-4c68-8974-c497b27fc517" (UID: "b20b4442-c4f3-4c68-8974-c497b27fc517"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:55:28.663686 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.663623 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-xks-ca" pod "b20b4442-c4f3-4c68-8974-c497b27fc517" (UID: "b20b4442-c4f3-4c68-8974-c497b27fc517"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:55:28.663791 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.663719 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20b4442-c4f3-4c68-8974-c497b27fc517-kube-api-access-r47zj" pod "b20b4442-c4f3-4c68-8974-c497b27fc517" (UID: "b20b4442-c4f3-4c68-8974-c497b27fc517"). InnerVolumeSpecName "kube-api-access-r47zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:55:28.762658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.762636 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:55:28.762658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.762659 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r47zj\" (UniqueName: \"kubernetes.io/projected/b20b4442-c4f3-4c68-8974-c497b27fc517-kube-api-access-r47zj\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:55:28.762798 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:28.762669 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20b4442-c4f3-4c68-8974-c497b27fc517-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:55:29.419022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:29.418991 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" Oct 30 14:55:29.419292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:29.419012 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz" event={"ID":"b20b4442-c4f3-4c68-8974-c497b27fc517","Type":"ContainerDied","Data":"75c85ac1109d1e8ac0c5dd5909bbd90732e40ffbb0490dc55f40bd7e34ccb598"} Oct 30 14:55:29.419292 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:29.419067 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75c85ac1109d1e8ac0c5dd5909bbd90732e40ffbb0490dc55f40bd7e34ccb598" Oct 30 14:55:33.052531 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:33.052499 5824 scope.go:122] "RemoveContainer" containerID="73b758e562477aa3c447bb37deb3c66a8af01eb7c4a2dd45a082b8e2d1ae9107" Oct 30 14:55:33.061304 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:33.061284 5824 scope.go:122] "RemoveContainer" containerID="241e3e99acdddc893166b602dcfb33441e7a113aa16cc397d240e0915efd6794" Oct 30 14:55:33.069225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:33.069182 5824 scope.go:122] "RemoveContainer" containerID="bc18c088da8a0c3caa3f741b063ad44728355115dc47a48089700d6321f57f79" Oct 30 14:55:33.076745 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:33.076729 5824 scope.go:122] "RemoveContainer" containerID="2a48b6b33f5f44857021bafb507609f5bcd7ace18b03632a2393f1156a5f9c26" Oct 30 14:55:33.641858 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:55:33.641823 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:55:40.692618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:40.692583 5824 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:55:40.693264 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:40.693229 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d7k5x" podUID="2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:55:42.512569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.512529 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg"] Oct 30 14:55:42.513016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.512977 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b20b4442-c4f3-4c68-8974-c497b27fc517" containerName="revoke-keys" Oct 30 14:55:42.513016 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.512991 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20b4442-c4f3-4c68-8974-c497b27fc517" containerName="revoke-keys" Oct 30 14:55:42.513107 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.513077 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20b4442-c4f3-4c68-8974-c497b27fc517" containerName="revoke-keys" Oct 30 14:55:42.516366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.516340 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.542871 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.542839 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:55:42.555789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.555751 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:55:42.555916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.555812 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:55:42.555916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.555809 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:55:42.558055 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.558034 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg"] Oct 30 14:55:42.608801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.608775 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb7bp\" (UniqueName: \"kubernetes.io/projected/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-kube-api-access-jb7bp\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.608929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.608821 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.608929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.608917 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.710070 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.710048 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb7bp\" (UniqueName: \"kubernetes.io/projected/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-kube-api-access-jb7bp\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.710223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.710085 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.710223 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.710115 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.710735 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.710712 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.712413 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.712391 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.736275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.736252 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb7bp\" (UniqueName: \"kubernetes.io/projected/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-kube-api-access-jb7bp\") pod \"maas-api-revoke-sub-0b7e8edb-qpjwg\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:42.830592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:42.830542 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:43.129254 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:43.129223 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg"] Oct 30 14:55:43.129533 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:55:43.129504 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fce53e3_5e81_4e21_89f3_f3e3d87e92d8.slice/crio-979f2133f9b2cba37cf64d421050fc83a2f19a752696e182e1ac5242382fcf03 WatchSource:0}: Error finding container 979f2133f9b2cba37cf64d421050fc83a2f19a752696e182e1ac5242382fcf03: Status 404 returned error can't find the container with id 979f2133f9b2cba37cf64d421050fc83a2f19a752696e182e1ac5242382fcf03 Oct 30 14:55:43.523233 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:43.523205 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" event={"ID":"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8","Type":"ContainerStarted","Data":"82dae7a7ae2ab21581359eabb6da399f59d6f9319a903ae9419e171b64b6339d"} Oct 30 14:55:43.523571 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:43.523278 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" event={"ID":"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8","Type":"ContainerStarted","Data":"979f2133f9b2cba37cf64d421050fc83a2f19a752696e182e1ac5242382fcf03"} Oct 30 14:55:43.567995 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:43.567936 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" podStartSLOduration=1.567924669 podStartE2EDuration="1.567924669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:43.56730127 +0000 UTC m=+3973.367806713" watchObservedRunningTime="2026-10-30 14:55:43.567924669 +0000 UTC m=+3973.368430105" Oct 30 14:55:43.913345 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:55:43.913320 5824 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fce53e3_5e81_4e21_89f3_f3e3d87e92d8.slice/crio-conmon-82dae7a7ae2ab21581359eabb6da399f59d6f9319a903ae9419e171b64b6339d.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:55:44.530545 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:44.530506 5824 generic.go:356] "Generic (PLEG): container finished" podID="1fce53e3-5e81-4e21-89f3-f3e3d87e92d8" containerID="82dae7a7ae2ab21581359eabb6da399f59d6f9319a903ae9419e171b64b6339d" exitCode=0 Oct 30 14:55:44.530926 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:44.530572 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" event={"ID":"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8","Type":"ContainerDied","Data":"82dae7a7ae2ab21581359eabb6da399f59d6f9319a903ae9419e171b64b6339d"} Oct 30 14:55:45.678983 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.678960 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:45.739505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.739480 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-openshift-ca\") pod \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " Oct 30 14:55:45.739638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.739536 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-kube-api-access-jb7bp\" (UniqueName: \"kubernetes.io/projected/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-kube-api-access-jb7bp\") pod \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " Oct 30 14:55:45.739638 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.739562 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-xks-ca\") pod \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\" (UID: \"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8\") " Oct 30 14:55:45.739868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.739844 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-openshift-ca" pod "1fce53e3-5e81-4e21-89f3-f3e3d87e92d8" (UID: "1fce53e3-5e81-4e21-89f3-f3e3d87e92d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:55:45.741532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.741513 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-kube-api-access-jb7bp" pod "1fce53e3-5e81-4e21-89f3-f3e3d87e92d8" (UID: "1fce53e3-5e81-4e21-89f3-f3e3d87e92d8"). InnerVolumeSpecName "kube-api-access-jb7bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:55:45.741639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.741614 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-xks-ca" pod "1fce53e3-5e81-4e21-89f3-f3e3d87e92d8" (UID: "1fce53e3-5e81-4e21-89f3-f3e3d87e92d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:55:45.841092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.841036 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:55:45.841092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.841059 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb7bp\" (UniqueName: \"kubernetes.io/projected/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-kube-api-access-jb7bp\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:55:45.841092 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:45.841070 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:55:46.545685 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:46.545658 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" Oct 30 14:55:46.545827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:46.545687 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg" event={"ID":"1fce53e3-5e81-4e21-89f3-f3e3d87e92d8","Type":"ContainerDied","Data":"979f2133f9b2cba37cf64d421050fc83a2f19a752696e182e1ac5242382fcf03"} Oct 30 14:55:46.545827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:55:46.545744 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979f2133f9b2cba37cf64d421050fc83a2f19a752696e182e1ac5242382fcf03" Oct 30 14:56:03.307599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.307555 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n"] Oct 30 14:56:03.308290 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.308268 5824 cpu_manager.go:414] "containerMap: removing container" podUID="1fce53e3-5e81-4e21-89f3-f3e3d87e92d8" containerName="revoke-keys" Oct 30 14:56:03.308392 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.308291 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fce53e3-5e81-4e21-89f3-f3e3d87e92d8" containerName="revoke-keys" Oct 30 14:56:03.308452 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.308402 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fce53e3-5e81-4e21-89f3-f3e3d87e92d8" containerName="revoke-keys" Oct 30 14:56:03.312266 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.312238 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.328670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.328643 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n"] Oct 30 14:56:03.340204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.340153 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:56:03.340204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.340153 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:56:03.340204 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.340162 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:56:03.342604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.342588 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:56:03.384658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.384633 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.384831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.384672 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47ptd\" (UniqueName: \"kubernetes.io/projected/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-kube-api-access-47ptd\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.384831 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.384773 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.485356 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.485329 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.485478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.485423 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.485478 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.485467 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ptd\" (UniqueName: \"kubernetes.io/projected/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-kube-api-access-47ptd\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.486144 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.486114 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.487768 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.487743 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.511269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.511244 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ptd\" (UniqueName: \"kubernetes.io/projected/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-kube-api-access-47ptd\") pod \"maas-api-revoke-sub-3d4156f1-9st9n\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.626065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.626012 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:03.642886 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:56:03.642856 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:56:03.929472 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:03.929445 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n"] Oct 30 14:56:03.930006 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:56:03.929983 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aa5ead_f2dd_4d3c_812c_c1df5cae016a.slice/crio-d7300dd9338f3b2df9c56594fe78f294734a24ec0cd4efe08a0a51a2318754e5 WatchSource:0}: Error finding container d7300dd9338f3b2df9c56594fe78f294734a24ec0cd4efe08a0a51a2318754e5: Status 404 returned error can't find the container with id d7300dd9338f3b2df9c56594fe78f294734a24ec0cd4efe08a0a51a2318754e5 Oct 30 14:56:04.651551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:04.651484 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" event={"ID":"84aa5ead-f2dd-4d3c-812c-c1df5cae016a","Type":"ContainerStarted","Data":"3012dbcb1e99f3c8d55de594a674178d3ff4550f25df5429ecbb1b00cf2e3e5f"} Oct 30 14:56:04.651551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:04.651543 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" event={"ID":"84aa5ead-f2dd-4d3c-812c-c1df5cae016a","Type":"ContainerStarted","Data":"d7300dd9338f3b2df9c56594fe78f294734a24ec0cd4efe08a0a51a2318754e5"} Oct 30 14:56:04.696319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:04.696271 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" podStartSLOduration=1.696252215 podStartE2EDuration="1.696252215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:56:04.695317162 +0000 UTC m=+3994.495822605" watchObservedRunningTime="2026-10-30 14:56:04.696252215 +0000 UTC m=+3994.496757658" Oct 30 14:56:05.659142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:05.659110 5824 generic.go:356] "Generic (PLEG): container finished" podID="84aa5ead-f2dd-4d3c-812c-c1df5cae016a" containerID="3012dbcb1e99f3c8d55de594a674178d3ff4550f25df5429ecbb1b00cf2e3e5f" exitCode=0 Oct 30 14:56:05.661672 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:05.659155 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" event={"ID":"84aa5ead-f2dd-4d3c-812c-c1df5cae016a","Type":"ContainerDied","Data":"3012dbcb1e99f3c8d55de594a674178d3ff4550f25df5429ecbb1b00cf2e3e5f"} Oct 30 14:56:06.782379 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:06.782354 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:06.917022 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:06.916960 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-openshift-ca\") pod \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " Oct 30 14:56:06.917142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:06.917021 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-xks-ca\") pod \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " Oct 30 14:56:06.917142 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:06.917049 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-kube-api-access-47ptd\" (UniqueName: \"kubernetes.io/projected/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-kube-api-access-47ptd\") pod \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\" (UID: \"84aa5ead-f2dd-4d3c-812c-c1df5cae016a\") " Oct 30 14:56:06.917417 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:06.917391 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-openshift-ca" pod "84aa5ead-f2dd-4d3c-812c-c1df5cae016a" (UID: "84aa5ead-f2dd-4d3c-812c-c1df5cae016a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:56:06.919072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:06.919049 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-xks-ca" pod "84aa5ead-f2dd-4d3c-812c-c1df5cae016a" (UID: "84aa5ead-f2dd-4d3c-812c-c1df5cae016a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:56:06.919147 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:06.919127 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-kube-api-access-47ptd" pod "84aa5ead-f2dd-4d3c-812c-c1df5cae016a" (UID: "84aa5ead-f2dd-4d3c-812c-c1df5cae016a"). InnerVolumeSpecName "kube-api-access-47ptd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:56:07.018162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:07.018135 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:56:07.018162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:07.018158 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47ptd\" (UniqueName: \"kubernetes.io/projected/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-kube-api-access-47ptd\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:56:07.018323 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:07.018167 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84aa5ead-f2dd-4d3c-812c-c1df5cae016a-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:56:07.672529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:07.672501 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" Oct 30 14:56:07.672697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:07.672535 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n" event={"ID":"84aa5ead-f2dd-4d3c-812c-c1df5cae016a","Type":"ContainerDied","Data":"d7300dd9338f3b2df9c56594fe78f294734a24ec0cd4efe08a0a51a2318754e5"} Oct 30 14:56:07.672697 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:07.672584 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7300dd9338f3b2df9c56594fe78f294734a24ec0cd4efe08a0a51a2318754e5" Oct 30 14:56:24.680592 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.680552 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n"] Oct 30 14:56:24.695181 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.695146 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9st9n"] Oct 30 14:56:24.731288 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.731261 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld"] Oct 30 14:56:24.731690 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.731676 5824 cpu_manager.go:414] "containerMap: removing container" podUID="84aa5ead-f2dd-4d3c-812c-c1df5cae016a" containerName="revoke-keys" Oct 30 14:56:24.731734 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.731691 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84aa5ead-f2dd-4d3c-812c-c1df5cae016a" containerName="revoke-keys" Oct 30 14:56:24.731790 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.731781 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="84aa5ead-f2dd-4d3c-812c-c1df5cae016a" containerName="revoke-keys" Oct 30 14:56:24.735269 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.735182 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.756131 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.756102 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld"] Oct 30 14:56:24.756444 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.756420 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:56:24.756604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.756425 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:56:24.756604 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.756464 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:56:24.766229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.766208 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:56:24.795762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.795734 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfsz\" (UniqueName: \"kubernetes.io/projected/a5fca84b-6b02-4a06-8105-3930b299252f-kube-api-access-8kfsz\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.795868 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.795793 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.795929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.795911 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.896777 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.896756 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.896894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.896794 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfsz\" (UniqueName: \"kubernetes.io/projected/a5fca84b-6b02-4a06-8105-3930b299252f-kube-api-access-8kfsz\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.896894 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.896830 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.897532 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.897507 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.899178 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.899155 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:24.923121 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:24.923088 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfsz\" (UniqueName: \"kubernetes.io/projected/a5fca84b-6b02-4a06-8105-3930b299252f-kube-api-access-8kfsz\") pod \"maas-api-revoke-sub-3d4156f1-nmcld\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:25.048224 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:25.048156 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:25.345305 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:25.345274 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld"] Oct 30 14:56:25.345495 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:56:25.345473 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fca84b_6b02_4a06_8105_3930b299252f.slice/crio-e7983b5c6b082cfe34c4e270f429299b63cef2f99bf6634f608a68a9f212e41d WatchSource:0}: Error finding container e7983b5c6b082cfe34c4e270f429299b63cef2f99bf6634f608a68a9f212e41d: Status 404 returned error can't find the container with id e7983b5c6b082cfe34c4e270f429299b63cef2f99bf6634f608a68a9f212e41d Oct 30 14:56:25.784636 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:25.784602 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" event={"ID":"a5fca84b-6b02-4a06-8105-3930b299252f","Type":"ContainerStarted","Data":"add9503e6a07ef005b87484d5d3cf14a3bce2eb3109daeefd165cb52089e9f57"} Oct 30 14:56:25.784990 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:25.784665 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" event={"ID":"a5fca84b-6b02-4a06-8105-3930b299252f","Type":"ContainerStarted","Data":"e7983b5c6b082cfe34c4e270f429299b63cef2f99bf6634f608a68a9f212e41d"} Oct 30 14:56:25.829167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:25.829127 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" podStartSLOduration=1.829116118 podStartE2EDuration="1.829116118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:56:25.828969279 +0000 UTC m=+4015.629474722" watchObservedRunningTime="2026-10-30 14:56:25.829116118 +0000 UTC m=+4015.629621561" Oct 30 14:56:26.315342 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:26.315315 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84aa5ead-f2dd-4d3c-812c-c1df5cae016a" path="/var/lib/kubelet/pods/84aa5ead-f2dd-4d3c-812c-c1df5cae016a/volumes" Oct 30 14:56:26.790945 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:26.790913 5824 generic.go:356] "Generic (PLEG): container finished" podID="a5fca84b-6b02-4a06-8105-3930b299252f" containerID="add9503e6a07ef005b87484d5d3cf14a3bce2eb3109daeefd165cb52089e9f57" exitCode=0 Oct 30 14:56:26.791404 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:26.790947 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" event={"ID":"a5fca84b-6b02-4a06-8105-3930b299252f","Type":"ContainerDied","Data":"add9503e6a07ef005b87484d5d3cf14a3bce2eb3109daeefd165cb52089e9f57"} Oct 30 14:56:27.923312 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:27.923289 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:28.024500 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.024472 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-openshift-ca\") pod \"a5fca84b-6b02-4a06-8105-3930b299252f\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " Oct 30 14:56:28.024649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.024535 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-xks-ca\") pod \"a5fca84b-6b02-4a06-8105-3930b299252f\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " Oct 30 14:56:28.024649 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.024617 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fca84b-6b02-4a06-8105-3930b299252f-kube-api-access-8kfsz\" (UniqueName: \"kubernetes.io/projected/a5fca84b-6b02-4a06-8105-3930b299252f-kube-api-access-8kfsz\") pod \"a5fca84b-6b02-4a06-8105-3930b299252f\" (UID: \"a5fca84b-6b02-4a06-8105-3930b299252f\") " Oct 30 14:56:28.024804 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.024780 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-openshift-ca" pod "a5fca84b-6b02-4a06-8105-3930b299252f" (UID: "a5fca84b-6b02-4a06-8105-3930b299252f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:56:28.024916 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.024901 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:56:28.026551 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.026527 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-xks-ca" pod "a5fca84b-6b02-4a06-8105-3930b299252f" (UID: "a5fca84b-6b02-4a06-8105-3930b299252f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:56:28.026656 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.026638 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fca84b-6b02-4a06-8105-3930b299252f-kube-api-access-8kfsz" pod "a5fca84b-6b02-4a06-8105-3930b299252f" (UID: "a5fca84b-6b02-4a06-8105-3930b299252f"). InnerVolumeSpecName "kube-api-access-8kfsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:56:28.048924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.048852 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f"] Oct 30 14:56:28.048924 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.048909 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8c52f"] Oct 30 14:56:28.125705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.125672 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fca84b-6b02-4a06-8105-3930b299252f-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:56:28.125705 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.125702 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kfsz\" (UniqueName: \"kubernetes.io/projected/a5fca84b-6b02-4a06-8105-3930b299252f-kube-api-access-8kfsz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:56:28.315460 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.315392 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b982f2-9323-4aec-83a5-63ecaf23463c" path="/var/lib/kubelet/pods/33b982f2-9323-4aec-83a5-63ecaf23463c/volumes" Oct 30 14:56:28.803019 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.802995 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" Oct 30 14:56:28.803216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.803032 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld" event={"ID":"a5fca84b-6b02-4a06-8105-3930b299252f","Type":"ContainerDied","Data":"e7983b5c6b082cfe34c4e270f429299b63cef2f99bf6634f608a68a9f212e41d"} Oct 30 14:56:28.803216 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:28.803081 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7983b5c6b082cfe34c4e270f429299b63cef2f99bf6634f608a68a9f212e41d" Oct 30 14:56:33.107964 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:33.107932 5824 scope.go:122] "RemoveContainer" containerID="0e79bcc8114bb8a9ed9dab3f9a8779550d2e4f6b1eaa9182a1c585abaf100aed" Oct 30 14:56:33.118776 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:33.118674 5824 scope.go:122] "RemoveContainer" containerID="1e5e5a6abd5a248977fab99fc6f7fdc987634fb23c4e7cf337ae9a595e101e83" Oct 30 14:56:33.126402 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:33.126384 5824 scope.go:122] "RemoveContainer" containerID="bc0cfc5fdc752aee0089e5067f4e330833ce6f0b4942009fdbf2dbf48a0dda62" Oct 30 14:56:33.134031 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:33.134005 5824 scope.go:122] "RemoveContainer" containerID="5030c2315aa0003c96d905690942c6deca34b888c36b11bba91570a78d5545e4" Oct 30 14:56:33.141099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:33.141082 5824 scope.go:122] "RemoveContainer" containerID="d7a3a59a225de34b3b5df735523ae7a6f105a71bb462e394b807d7142c141563" Oct 30 14:56:33.147910 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:33.147894 5824 scope.go:122] "RemoveContainer" containerID="29acc643cd8a82fb8b845b28fb6541e37c34a60f8665e4de4deb2af94f04bb8b" Oct 30 14:56:33.154818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:33.154801 5824 scope.go:122] "RemoveContainer" containerID="31c5eec888fba258f11b264e313bd59aecd978076f7fef4664b286ed73769d38" Oct 30 14:56:33.161787 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:33.161771 5824 scope.go:122] "RemoveContainer" containerID="acc1d739e0e818bfd8f8752e3221eab219dbe9668d50131b8987e81d1d5d81ac" Oct 30 14:56:33.640009 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:56:33.639969 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:56:58.639730 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.639696 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld"] Oct 30 14:56:58.652675 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.652643 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nmcld"] Oct 30 14:56:58.695976 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.695946 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7"] Oct 30 14:56:58.696377 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.696363 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a5fca84b-6b02-4a06-8105-3930b299252f" containerName="revoke-keys" Oct 30 14:56:58.696442 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.696378 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fca84b-6b02-4a06-8105-3930b299252f" containerName="revoke-keys" Oct 30 14:56:58.696488 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.696469 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fca84b-6b02-4a06-8105-3930b299252f" containerName="revoke-keys" Oct 30 14:56:58.699747 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.699728 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.717075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.717048 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:56:58.717249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.717159 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:56:58.717249 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.717210 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:56:58.717405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.717330 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:56:58.718466 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.718440 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7"] Oct 30 14:56:58.804405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.804366 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.804405 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.804406 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzfb\" (UniqueName: \"kubernetes.io/projected/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-kube-api-access-lpzfb\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.804674 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.804523 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.905067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.904982 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.905067 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.905019 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzfb\" (UniqueName: \"kubernetes.io/projected/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-kube-api-access-lpzfb\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.905313 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.905073 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.905748 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.905725 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.907464 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.907437 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:58.931561 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:58.931531 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzfb\" (UniqueName: \"kubernetes.io/projected/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-kube-api-access-lpzfb\") pod \"maas-api-revoke-sub-3d4156f1-rq9b7\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:59.012980 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:59.012945 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:56:59.313940 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:59.311902 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7"] Oct 30 14:56:59.316652 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:56:59.316619 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebdd4be_0798_46c9_a7d7_c4f57c72cbab.slice/crio-66a0b692920d62ea12842702b0c82874031a15a6d4ae647a5c779498c54437a3 WatchSource:0}: Error finding container 66a0b692920d62ea12842702b0c82874031a15a6d4ae647a5c779498c54437a3: Status 404 returned error can't find the container with id 66a0b692920d62ea12842702b0c82874031a15a6d4ae647a5c779498c54437a3 Oct 30 14:56:59.959425 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:59.959388 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" event={"ID":"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab","Type":"ContainerStarted","Data":"94effb9ba8ebb6b90c60a8ae38704b750eec2ddbb55515c6396be9a27199e106"} Oct 30 14:56:59.959818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:56:59.959456 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" event={"ID":"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab","Type":"ContainerStarted","Data":"66a0b692920d62ea12842702b0c82874031a15a6d4ae647a5c779498c54437a3"} Oct 30 14:57:00.005428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:00.005375 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" podStartSLOduration=2.005362143 podStartE2EDuration="2.005362143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:57:00.003817462 +0000 UTC m=+4049.804322897" watchObservedRunningTime="2026-10-30 14:57:00.005362143 +0000 UTC m=+4049.805867586" Oct 30 14:57:00.315688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:00.315607 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fca84b-6b02-4a06-8105-3930b299252f" path="/var/lib/kubelet/pods/a5fca84b-6b02-4a06-8105-3930b299252f/volumes" Oct 30 14:57:00.966093 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:00.966059 5824 generic.go:356] "Generic (PLEG): container finished" podID="4ebdd4be-0798-46c9-a7d7-c4f57c72cbab" containerID="94effb9ba8ebb6b90c60a8ae38704b750eec2ddbb55515c6396be9a27199e106" exitCode=0 Oct 30 14:57:00.966591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:00.966157 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" event={"ID":"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab","Type":"ContainerDied","Data":"94effb9ba8ebb6b90c60a8ae38704b750eec2ddbb55515c6396be9a27199e106"} Oct 30 14:57:02.104937 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.104908 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:57:02.137769 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.137734 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-kube-api-access-lpzfb\" (UniqueName: \"kubernetes.io/projected/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-kube-api-access-lpzfb\") pod \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " Oct 30 14:57:02.137975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.137800 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-openshift-ca\") pod \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " Oct 30 14:57:02.137975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.137835 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-xks-ca\") pod \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\" (UID: \"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab\") " Oct 30 14:57:02.138183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.138160 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-openshift-ca" pod "4ebdd4be-0798-46c9-a7d7-c4f57c72cbab" (UID: "4ebdd4be-0798-46c9-a7d7-c4f57c72cbab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:57:02.139966 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.139941 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-xks-ca" pod "4ebdd4be-0798-46c9-a7d7-c4f57c72cbab" (UID: "4ebdd4be-0798-46c9-a7d7-c4f57c72cbab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:57:02.140072 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.140013 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-kube-api-access-lpzfb" pod "4ebdd4be-0798-46c9-a7d7-c4f57c72cbab" (UID: "4ebdd4be-0798-46c9-a7d7-c4f57c72cbab"). InnerVolumeSpecName "kube-api-access-lpzfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:57:02.238640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.238556 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:57:02.238640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.238589 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpzfb\" (UniqueName: \"kubernetes.io/projected/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-kube-api-access-lpzfb\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:57:02.238640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.238600 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:57:02.977528 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.977497 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" Oct 30 14:57:02.977714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.977531 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7" event={"ID":"4ebdd4be-0798-46c9-a7d7-c4f57c72cbab","Type":"ContainerDied","Data":"66a0b692920d62ea12842702b0c82874031a15a6d4ae647a5c779498c54437a3"} Oct 30 14:57:02.977714 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:02.977585 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a0b692920d62ea12842702b0c82874031a15a6d4ae647a5c779498c54437a3" Oct 30 14:57:03.641624 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:57:03.641585 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:57:14.485827 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.485553 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx"] Oct 30 14:57:14.486331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.486017 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4ebdd4be-0798-46c9-a7d7-c4f57c72cbab" containerName="revoke-keys" Oct 30 14:57:14.486331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.486033 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebdd4be-0798-46c9-a7d7-c4f57c72cbab" containerName="revoke-keys" Oct 30 14:57:14.486331 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.486179 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebdd4be-0798-46c9-a7d7-c4f57c72cbab" containerName="revoke-keys" Oct 30 14:57:14.489409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.489390 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.513512 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.513481 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx"] Oct 30 14:57:14.520750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.520722 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d3d611-openshift-default-dockercfg-fvvgs\"" Oct 30 14:57:14.553752 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.553724 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-socket\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.553865 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.553759 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-credential-socket\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.553912 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.553868 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93e9207-0517-44a4-a0ec-0233acd50963-istio-podinfo\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.553953 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.553926 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnrwg\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-kube-api-access-rnrwg\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.554013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.553957 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-certs\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.554013 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.553986 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-envoy\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.554135 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.554108 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-istio-token\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.554225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.554165 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93e9207-0517-44a4-a0ec-0233acd50963-istiod-ca-cert\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.554294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.554232 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-data\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655200 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655154 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-istio-token\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655437 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655411 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93e9207-0517-44a4-a0ec-0233acd50963-istiod-ca-cert\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655556 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655454 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-data\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655571 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-socket\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655640 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655608 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-credential-socket\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655760 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655688 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93e9207-0517-44a4-a0ec-0233acd50963-istio-podinfo\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655760 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655724 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnrwg\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-kube-api-access-rnrwg\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655867 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655765 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-certs\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.655867 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.655813 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-envoy\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.656183 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.656152 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93e9207-0517-44a4-a0ec-0233acd50963-istiod-ca-cert\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.656427 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.656405 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-credential-socket\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.656687 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.656665 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-data\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.656946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.656923 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-socket\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.657021 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.656970 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-certs\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.658552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.658527 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-envoy\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.659399 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.659373 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93e9207-0517-44a4-a0ec-0233acd50963-istio-podinfo\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.682368 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.682335 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-istio-token\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.682606 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.682380 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnrwg\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-kube-api-access-rnrwg\") pod \"e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:14.801708 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:14.801647 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:15.117658 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:15.117629 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx"] Oct 30 14:57:15.117991 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:57:15.117968 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93e9207_0517_44a4_a0ec_0233acd50963.slice/crio-d816f55a404ab2709718123bc39c22f1302aeec713537f3d6bffc9e8eeb1b577 WatchSource:0}: Error finding container d816f55a404ab2709718123bc39c22f1302aeec713537f3d6bffc9e8eeb1b577: Status 404 returned error can't find the container with id d816f55a404ab2709718123bc39c22f1302aeec713537f3d6bffc9e8eeb1b577 Oct 30 14:57:15.120057 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:15.120022 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:57:15.120148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:15.120087 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:57:15.120148 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:15.120118 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:57:16.046023 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:16.045987 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" event={"ID":"a93e9207-0517-44a4-a0ec-0233acd50963","Type":"ContainerStarted","Data":"b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca"} Oct 30 14:57:16.046431 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:16.046045 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" event={"ID":"a93e9207-0517-44a4-a0ec-0233acd50963","Type":"ContainerStarted","Data":"d816f55a404ab2709718123bc39c22f1302aeec713537f3d6bffc9e8eeb1b577"} Oct 30 14:57:16.110593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:16.110544 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" podStartSLOduration=2.110529781 podStartE2EDuration="2.110529781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:57:16.108735295 +0000 UTC m=+4065.909240738" watchObservedRunningTime="2026-10-30 14:57:16.110529781 +0000 UTC m=+4065.911035288" Oct 30 14:57:16.801883 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:16.801850 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:16.806750 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:16.806723 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:17.053253 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:17.053029 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:17.054884 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:17.054852 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:20.437432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.437402 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886"] Oct 30 14:57:20.440917 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.440895 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:20.460268 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.460242 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886"] Oct 30 14:57:20.471670 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.471643 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d3d611\"" Oct 30 14:57:20.471789 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.471733 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-13b67c74-dockercfg-w99st\"" Oct 30 14:57:20.513539 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.513507 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqh98\" (UniqueName: \"kubernetes.io/projected/6955f413-b60c-4f55-b7f2-a72b9b497499-kube-api-access-rqh98\") pod \"payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886\" (UID: \"6955f413-b60c-4f55-b7f2-a72b9b497499\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:20.513721 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.513603 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6955f413-b60c-4f55-b7f2-a72b9b497499-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886\" (UID: \"6955f413-b60c-4f55-b7f2-a72b9b497499\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:20.516975 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.516945 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n"] Oct 30 14:57:20.521578 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.521554 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:20.542744 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.542718 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n"] Oct 30 14:57:20.614935 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.614900 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqh98\" (UniqueName: \"kubernetes.io/projected/6955f413-b60c-4f55-b7f2-a72b9b497499-kube-api-access-rqh98\") pod \"payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886\" (UID: \"6955f413-b60c-4f55-b7f2-a72b9b497499\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:20.615100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.614983 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrl9\" (UniqueName: \"kubernetes.io/projected/719c4136-7e7a-4d4d-92e0-c81510a6904e-kube-api-access-whrl9\") pod \"payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n\" (UID: \"719c4136-7e7a-4d4d-92e0-c81510a6904e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:20.615100 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.615057 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6955f413-b60c-4f55-b7f2-a72b9b497499-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886\" (UID: \"6955f413-b60c-4f55-b7f2-a72b9b497499\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:20.615236 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.615133 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719c4136-7e7a-4d4d-92e0-c81510a6904e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n\" (UID: \"719c4136-7e7a-4d4d-92e0-c81510a6904e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:20.615780 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.615762 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6955f413-b60c-4f55-b7f2-a72b9b497499-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886\" (UID: \"6955f413-b60c-4f55-b7f2-a72b9b497499\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:20.641946 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.641922 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqh98\" (UniqueName: \"kubernetes.io/projected/6955f413-b60c-4f55-b7f2-a72b9b497499-kube-api-access-rqh98\") pod \"payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886\" (UID: \"6955f413-b60c-4f55-b7f2-a72b9b497499\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:20.715811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.715745 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrl9\" (UniqueName: \"kubernetes.io/projected/719c4136-7e7a-4d4d-92e0-c81510a6904e-kube-api-access-whrl9\") pod \"payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n\" (UID: \"719c4136-7e7a-4d4d-92e0-c81510a6904e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:20.715811 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.715799 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719c4136-7e7a-4d4d-92e0-c81510a6904e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n\" (UID: \"719c4136-7e7a-4d4d-92e0-c81510a6904e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:20.716506 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.716487 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719c4136-7e7a-4d4d-92e0-c81510a6904e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n\" (UID: \"719c4136-7e7a-4d4d-92e0-c81510a6904e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:20.742102 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.742075 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrl9\" (UniqueName: \"kubernetes.io/projected/719c4136-7e7a-4d4d-92e0-c81510a6904e-kube-api-access-whrl9\") pod \"payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n\" (UID: \"719c4136-7e7a-4d4d-92e0-c81510a6904e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:20.753552 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.753530 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:20.837719 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:20.837688 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:21.053262 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:21.053232 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886"] Oct 30 14:57:21.053439 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:57:21.053410 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6955f413_b60c_4f55_b7f2_a72b9b497499.slice/crio-0a1247c466c8bbefbf8b415050804be1e5c667ec192a89a8c5ef3fea12a63e99 WatchSource:0}: Error finding container 0a1247c466c8bbefbf8b415050804be1e5c667ec192a89a8c5ef3fea12a63e99: Status 404 returned error can't find the container with id 0a1247c466c8bbefbf8b415050804be1e5c667ec192a89a8c5ef3fea12a63e99 Oct 30 14:57:21.076919 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:21.076891 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" event={"ID":"6955f413-b60c-4f55-b7f2-a72b9b497499","Type":"ContainerStarted","Data":"0a1247c466c8bbefbf8b415050804be1e5c667ec192a89a8c5ef3fea12a63e99"} Oct 30 14:57:21.140097 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:57:21.140053 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719c4136_7e7a_4d4d_92e0_c81510a6904e.slice/crio-4ebfb101ec0502fb54cfef5bd7c6bae74a27a8ce7a83c3b41e16316921530800 WatchSource:0}: Error finding container 4ebfb101ec0502fb54cfef5bd7c6bae74a27a8ce7a83c3b41e16316921530800: Status 404 returned error can't find the container with id 4ebfb101ec0502fb54cfef5bd7c6bae74a27a8ce7a83c3b41e16316921530800 Oct 30 14:57:21.140818 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:21.140794 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n"] Oct 30 14:57:22.084045 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:22.084004 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" event={"ID":"719c4136-7e7a-4d4d-92e0-c81510a6904e","Type":"ContainerStarted","Data":"7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499"} Oct 30 14:57:22.084557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:22.084111 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:22.084557 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:22.084141 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" event={"ID":"719c4136-7e7a-4d4d-92e0-c81510a6904e","Type":"ContainerStarted","Data":"4ebfb101ec0502fb54cfef5bd7c6bae74a27a8ce7a83c3b41e16316921530800"} Oct 30 14:57:22.085662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:22.085634 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" event={"ID":"6955f413-b60c-4f55-b7f2-a72b9b497499","Type":"ContainerStarted","Data":"16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12"} Oct 30 14:57:22.085817 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:22.085797 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:22.133617 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:22.133567 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" podStartSLOduration=2.133554964 podStartE2EDuration="2.133554964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:57:22.132694846 +0000 UTC m=+4071.933200289" watchObservedRunningTime="2026-10-30 14:57:22.133554964 +0000 UTC m=+4071.934060408" Oct 30 14:57:22.179646 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:22.179603 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" podStartSLOduration=2.179591321 podStartE2EDuration="2.179591321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:57:22.178146232 +0000 UTC m=+4071.978651678" watchObservedRunningTime="2026-10-30 14:57:22.179591321 +0000 UTC m=+4071.980096766" Oct 30 14:57:33.094036 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:33.094002 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:57:33.094618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:33.094267 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:57:33.094618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:33.094286 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 14:57:33.094618 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:33.094328 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 14:57:33.204870 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:33.204845 5824 scope.go:122] "RemoveContainer" containerID="a439b1b813d2e1f75db56217c95412e2d68bab14843434d609a98f8a74f4e403" Oct 30 14:57:33.641081 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:57:33.641038 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:57:34.152522 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:34.152491 5824 generic.go:356] "Generic (PLEG): container finished" podID="a93e9207-0517-44a4-a0ec-0233acd50963" containerID="b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca" exitCode=0 Oct 30 14:57:34.152958 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:34.152566 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" event={"ID":"a93e9207-0517-44a4-a0ec-0233acd50963","Type":"ContainerDied","Data":"b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca"} Oct 30 14:57:34.153003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:34.152979 5824 scope.go:122] "RemoveContainer" containerID="b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca" Oct 30 14:57:34.154340 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:34.154305 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:57:34.154454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:34.154379 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:57:34.154454 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:34.154413 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 14:57:35.170753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:35.170713 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" event={"ID":"a93e9207-0517-44a4-a0ec-0233acd50963","Type":"ContainerStarted","Data":"2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9"} Oct 30 14:57:35.802126 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:35.802090 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:35.802429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:35.802398 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 14:57:35.802568 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:35.802458 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 14:57:36.802514 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:36.802478 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 14:57:36.802947 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:36.802528 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 14:57:37.036853 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:37.036824 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw"] Oct 30 14:57:37.063665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:37.063586 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lztnw"] Oct 30 14:57:37.803098 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:37.803053 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 14:57:37.803558 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:37.803114 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 14:57:38.315767 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:38.315727 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48834e4f-f5f0-43eb-b2ad-97e50fce68a5" path="/var/lib/kubelet/pods/48834e4f-f5f0-43eb-b2ad-97e50fce68a5/volumes" Oct 30 14:57:38.802906 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:38.802875 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 14:57:38.803101 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:38.802927 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 14:57:39.802762 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:39.802722 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 14:57:39.803222 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:39.802778 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 14:57:40.806113 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:40.806079 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:40.806591 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:40.806448 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:57:40.807212 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:57:40.807162 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:58:00.036530 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:00.036481 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns"] Oct 30 14:58:00.049833 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:00.049802 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-v24ns"] Oct 30 14:58:00.316429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:00.316356 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ccd6ed-8d41-433a-94ce-80251c3d3be6" path="/var/lib/kubelet/pods/d0ccd6ed-8d41-433a-94ce-80251c3d3be6/volumes" Oct 30 14:58:03.643047 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:58:03.642998 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:58:11.059688 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:11.059653 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj"] Oct 30 14:58:11.072421 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:11.072390 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-79hfj"] Oct 30 14:58:12.314760 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:12.314724 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc" path="/var/lib/kubelet/pods/ea8bf930-d23a-4d0e-9bc2-fbebcd4fa9bc/volumes" Oct 30 14:58:14.329003 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.328962 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws"] Oct 30 14:58:14.333075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.333054 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.351713 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.351665 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws"] Oct 30 14:58:14.362474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.362444 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:58:14.362586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.362451 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:58:14.362586 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.362451 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:58:14.363240 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.363226 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:58:14.434167 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.434144 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.434310 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.434206 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnt8\" (UniqueName: \"kubernetes.io/projected/07687d7b-87ee-4bef-949f-1e89f05b6a3f-kube-api-access-7cnt8\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.434366 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.434345 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.535344 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.535314 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.535476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.535365 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.535476 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.535412 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnt8\" (UniqueName: \"kubernetes.io/projected/07687d7b-87ee-4bef-949f-1e89f05b6a3f-kube-api-access-7cnt8\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.535954 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.535934 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.537766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.537736 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.561467 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.561442 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnt8\" (UniqueName: \"kubernetes.io/projected/07687d7b-87ee-4bef-949f-1e89f05b6a3f-kube-api-access-7cnt8\") pod \"maas-api-revoke-sub-5d41b8de-n5jws\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.645952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.645894 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:14.945572 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:14.945536 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws"] Oct 30 14:58:14.945889 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:58:14.945868 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07687d7b_87ee_4bef_949f_1e89f05b6a3f.slice/crio-e85bc2cc1d7240120c6bc1a40788f33cc4cb966cc163bd6d3731188aa3e5d25e WatchSource:0}: Error finding container e85bc2cc1d7240120c6bc1a40788f33cc4cb966cc163bd6d3731188aa3e5d25e: Status 404 returned error can't find the container with id e85bc2cc1d7240120c6bc1a40788f33cc4cb966cc163bd6d3731188aa3e5d25e Oct 30 14:58:15.389229 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:15.389185 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" event={"ID":"07687d7b-87ee-4bef-949f-1e89f05b6a3f","Type":"ContainerStarted","Data":"adc17b6f03c262260c8648b9851bf550a66585cb3549bca87f32856bcac765d6"} Oct 30 14:58:15.389599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:15.389242 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" event={"ID":"07687d7b-87ee-4bef-949f-1e89f05b6a3f","Type":"ContainerStarted","Data":"e85bc2cc1d7240120c6bc1a40788f33cc4cb966cc163bd6d3731188aa3e5d25e"} Oct 30 14:58:15.436550 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:15.436513 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" podStartSLOduration=1.436500532 podStartE2EDuration="1.436500532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:15.435687169 +0000 UTC m=+4125.236192613" watchObservedRunningTime="2026-10-30 14:58:15.436500532 +0000 UTC m=+4125.237005976" Oct 30 14:58:16.395256 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:16.395225 5824 generic.go:356] "Generic (PLEG): container finished" podID="07687d7b-87ee-4bef-949f-1e89f05b6a3f" containerID="adc17b6f03c262260c8648b9851bf550a66585cb3549bca87f32856bcac765d6" exitCode=0 Oct 30 14:58:16.395645 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:16.395309 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" event={"ID":"07687d7b-87ee-4bef-949f-1e89f05b6a3f","Type":"ContainerDied","Data":"adc17b6f03c262260c8648b9851bf550a66585cb3549bca87f32856bcac765d6"} Oct 30 14:58:17.541665 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.541635 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:17.665428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.665321 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-openshift-ca\") pod \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " Oct 30 14:58:17.665428 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.665416 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07687d7b-87ee-4bef-949f-1e89f05b6a3f-kube-api-access-7cnt8\" (UniqueName: \"kubernetes.io/projected/07687d7b-87ee-4bef-949f-1e89f05b6a3f-kube-api-access-7cnt8\") pod \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " Oct 30 14:58:17.665855 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.665504 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-xks-ca\") pod \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\" (UID: \"07687d7b-87ee-4bef-949f-1e89f05b6a3f\") " Oct 30 14:58:17.668248 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.668218 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-openshift-ca" pod "07687d7b-87ee-4bef-949f-1e89f05b6a3f" (UID: "07687d7b-87ee-4bef-949f-1e89f05b6a3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:58:17.668929 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.668902 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-xks-ca" pod "07687d7b-87ee-4bef-949f-1e89f05b6a3f" (UID: "07687d7b-87ee-4bef-949f-1e89f05b6a3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:58:17.670804 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.670765 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07687d7b-87ee-4bef-949f-1e89f05b6a3f-kube-api-access-7cnt8" pod "07687d7b-87ee-4bef-949f-1e89f05b6a3f" (UID: "07687d7b-87ee-4bef-949f-1e89f05b6a3f"). InnerVolumeSpecName "kube-api-access-7cnt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:58:17.766800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.766766 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:17.766800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.766795 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cnt8\" (UniqueName: \"kubernetes.io/projected/07687d7b-87ee-4bef-949f-1e89f05b6a3f-kube-api-access-7cnt8\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:17.766800 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:17.766805 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07687d7b-87ee-4bef-949f-1e89f05b6a3f-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:18.406429 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:18.406402 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" Oct 30 14:58:18.406576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:18.406438 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws" event={"ID":"07687d7b-87ee-4bef-949f-1e89f05b6a3f","Type":"ContainerDied","Data":"e85bc2cc1d7240120c6bc1a40788f33cc4cb966cc163bd6d3731188aa3e5d25e"} Oct 30 14:58:18.406576 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:18.406477 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e85bc2cc1d7240120c6bc1a40788f33cc4cb966cc163bd6d3731188aa3e5d25e" Oct 30 14:58:25.579682 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.579648 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw"] Oct 30 14:58:25.580146 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.580123 5824 cpu_manager.go:414] "containerMap: removing container" podUID="07687d7b-87ee-4bef-949f-1e89f05b6a3f" containerName="revoke-keys" Oct 30 14:58:25.580211 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.580148 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07687d7b-87ee-4bef-949f-1e89f05b6a3f" containerName="revoke-keys" Oct 30 14:58:25.580282 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.580270 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="07687d7b-87ee-4bef-949f-1e89f05b6a3f" containerName="revoke-keys" Oct 30 14:58:25.583652 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.583630 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.600778 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.600730 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw"] Oct 30 14:58:25.601309 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.601288 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:58:25.611851 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.611833 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:58:25.612241 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.612225 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:58:25.614403 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.614374 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:58:25.732659 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.732629 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.732779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.732673 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq49v\" (UniqueName: \"kubernetes.io/projected/0a287184-a0ee-4f78-a821-3954223e37f0-kube-api-access-kq49v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.732779 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.732732 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.833397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.833327 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq49v\" (UniqueName: \"kubernetes.io/projected/0a287184-a0ee-4f78-a821-3954223e37f0-kube-api-access-kq49v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.833397 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.833381 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.833582 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.833532 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.834125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.834104 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.835653 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.835625 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.859294 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.859269 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq49v\" (UniqueName: \"kubernetes.io/projected/0a287184-a0ee-4f78-a821-3954223e37f0-kube-api-access-kq49v\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:25.896409 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:25.896388 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:26.200082 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:26.200054 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw"] Oct 30 14:58:26.202040 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:58:26.202011 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a287184_a0ee_4f78_a821_3954223e37f0.slice/crio-666ef5c5cdb82afab4af1448a4ec4420cb18247dff154f5046ae53df47d2ec8f WatchSource:0}: Error finding container 666ef5c5cdb82afab4af1448a4ec4420cb18247dff154f5046ae53df47d2ec8f: Status 404 returned error can't find the container with id 666ef5c5cdb82afab4af1448a4ec4420cb18247dff154f5046ae53df47d2ec8f Oct 30 14:58:26.447487 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:26.447462 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" event={"ID":"0a287184-a0ee-4f78-a821-3954223e37f0","Type":"ContainerStarted","Data":"666ef5c5cdb82afab4af1448a4ec4420cb18247dff154f5046ae53df47d2ec8f"} Oct 30 14:58:27.453740 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:27.453709 5824 generic.go:356] "Generic (PLEG): container finished" podID="0a287184-a0ee-4f78-a821-3954223e37f0" containerID="d8e83b8e5cd8d90b457c15ad3a2d259d31413a05998c90450c208db781035c74" exitCode=0 Oct 30 14:58:27.454119 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:27.453792 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" event={"ID":"0a287184-a0ee-4f78-a821-3954223e37f0","Type":"ContainerDied","Data":"d8e83b8e5cd8d90b457c15ad3a2d259d31413a05998c90450c208db781035c74"} Oct 30 14:58:28.592799 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.592770 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:28.759634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.759551 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a287184-a0ee-4f78-a821-3954223e37f0-kube-api-access-kq49v\" (UniqueName: \"kubernetes.io/projected/0a287184-a0ee-4f78-a821-3954223e37f0-kube-api-access-kq49v\") pod \"0a287184-a0ee-4f78-a821-3954223e37f0\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " Oct 30 14:58:28.759634 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.759605 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-openshift-ca\") pod \"0a287184-a0ee-4f78-a821-3954223e37f0\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " Oct 30 14:58:28.759857 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.759739 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-xks-ca\") pod \"0a287184-a0ee-4f78-a821-3954223e37f0\" (UID: \"0a287184-a0ee-4f78-a821-3954223e37f0\") " Oct 30 14:58:28.759968 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.759946 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-openshift-ca" pod "0a287184-a0ee-4f78-a821-3954223e37f0" (UID: "0a287184-a0ee-4f78-a821-3954223e37f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:58:28.760125 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.760104 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:28.761438 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.761416 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a287184-a0ee-4f78-a821-3954223e37f0-kube-api-access-kq49v" pod "0a287184-a0ee-4f78-a821-3954223e37f0" (UID: "0a287184-a0ee-4f78-a821-3954223e37f0"). InnerVolumeSpecName "kube-api-access-kq49v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:58:28.761593 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.761571 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-xks-ca" pod "0a287184-a0ee-4f78-a821-3954223e37f0" (UID: "0a287184-a0ee-4f78-a821-3954223e37f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:58:28.861400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.861368 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq49v\" (UniqueName: \"kubernetes.io/projected/0a287184-a0ee-4f78-a821-3954223e37f0-kube-api-access-kq49v\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:28.861400 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:28.861399 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a287184-a0ee-4f78-a821-3954223e37f0-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:29.466414 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:29.466383 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" Oct 30 14:58:29.466585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:29.466393 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw" event={"ID":"0a287184-a0ee-4f78-a821-3954223e37f0","Type":"ContainerDied","Data":"666ef5c5cdb82afab4af1448a4ec4420cb18247dff154f5046ae53df47d2ec8f"} Oct 30 14:58:29.466585 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:29.466501 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666ef5c5cdb82afab4af1448a4ec4420cb18247dff154f5046ae53df47d2ec8f" Oct 30 14:58:33.640225 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:58:33.640170 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:58:36.026381 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:36.026348 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n"] Oct 30 14:58:36.060270 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:36.060238 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886"] Oct 30 14:58:37.557327 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:37.557290 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw"] Oct 30 14:58:37.571324 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:37.571294 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d3d611-tntmw"] Oct 30 14:58:38.316319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:38.316290 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a287184-a0ee-4f78-a821-3954223e37f0" path="/var/lib/kubelet/pods/0a287184-a0ee-4f78-a821-3954223e37f0/volumes" Oct 30 14:58:41.027185 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:41.027099 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" podUID="719c4136-7e7a-4d4d-92e0-c81510a6904e" containerName="payload-processing" containerID="cri-o://7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499" gracePeriod=25 Oct 30 14:58:41.061441 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:41.061408 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" podUID="6955f413-b60c-4f55-b7f2-a72b9b497499" containerName="payload-pre-processing" containerID="cri-o://16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12" gracePeriod=25 Oct 30 14:58:43.092862 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:43.092829 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" podUID="6955f413-b60c-4f55-b7f2-a72b9b497499" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:58:43.093285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:43.092934 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" podUID="719c4136-7e7a-4d4d-92e0-c81510a6904e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:58:44.950780 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:44.950741 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx"] Oct 30 14:58:44.951373 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:44.951020 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" containerID="cri-o://2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9" gracePeriod=30 Oct 30 14:58:46.223537 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.223498 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7"] Oct 30 14:58:46.236589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.236562 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rq9b7"] Oct 30 14:58:46.278372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.278349 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2"] Oct 30 14:58:46.278784 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.278751 5824 cpu_manager.go:414] "containerMap: removing container" podUID="0a287184-a0ee-4f78-a821-3954223e37f0" containerName="revoke-keys" Oct 30 14:58:46.278835 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.278787 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a287184-a0ee-4f78-a821-3954223e37f0" containerName="revoke-keys" Oct 30 14:58:46.278897 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.278888 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a287184-a0ee-4f78-a821-3954223e37f0" containerName="revoke-keys" Oct 30 14:58:46.282319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.282299 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.300250 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.300223 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 14:58:46.300384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.300267 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:58:46.300384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.300287 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 14:58:46.300384 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.300311 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:58:46.301024 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.301003 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2"] Oct 30 14:58:46.313900 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.313872 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.313998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.313955 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9s27\" (UniqueName: \"kubernetes.io/projected/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-kube-api-access-h9s27\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.314120 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.314101 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.315801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.315781 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebdd4be-0798-46c9-a7d7-c4f57c72cbab" path="/var/lib/kubelet/pods/4ebdd4be-0798-46c9-a7d7-c4f57c72cbab/volumes" Oct 30 14:58:46.415162 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.415143 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.415304 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.415177 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.415372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.415310 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9s27\" (UniqueName: \"kubernetes.io/projected/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-kube-api-access-h9s27\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.415854 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.415829 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.417474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.417451 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.441155 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.441135 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9s27\" (UniqueName: \"kubernetes.io/projected/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-kube-api-access-h9s27\") pod \"maas-api-revoke-sub-3d4156f1-j95p2\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.595482 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.595431 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:46.896520 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:46.896489 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2"] Oct 30 14:58:46.896645 ip-10-0-131-200 kubenswrapper[5824]: W1030 14:58:46.896577 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9687edf9_7a77_4f59_83e6_b9e87e4ce4df.slice/crio-57eb2feb802b7c6a888fbd2d4094074eaa615f3928b2193202bc630b0bce3ac8 WatchSource:0}: Error finding container 57eb2feb802b7c6a888fbd2d4094074eaa615f3928b2193202bc630b0bce3ac8: Status 404 returned error can't find the container with id 57eb2feb802b7c6a888fbd2d4094074eaa615f3928b2193202bc630b0bce3ac8 Oct 30 14:58:47.562923 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:47.562885 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" event={"ID":"9687edf9-7a77-4f59-83e6-b9e87e4ce4df","Type":"ContainerStarted","Data":"4745f99c433e1e62dc32a0c49eb71ddec271e25c75b79689e620f2628e827f60"} Oct 30 14:58:47.563346 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:47.562944 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" event={"ID":"9687edf9-7a77-4f59-83e6-b9e87e4ce4df","Type":"ContainerStarted","Data":"57eb2feb802b7c6a888fbd2d4094074eaa615f3928b2193202bc630b0bce3ac8"} Oct 30 14:58:47.608529 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:47.608460 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" podStartSLOduration=1.608441922 podStartE2EDuration="1.608441922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:47.607993351 +0000 UTC m=+4157.408498794" watchObservedRunningTime="2026-10-30 14:58:47.608441922 +0000 UTC m=+4157.408947367" Oct 30 14:58:48.040285 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:48.040252 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4"] Oct 30 14:58:48.044243 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:48.044220 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hq2t4"] Oct 30 14:58:48.316136 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:48.316060 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4973a1-f089-49ed-b1db-8eef28bc25ad" path="/var/lib/kubelet/pods/fd4973a1-f089-49ed-b1db-8eef28bc25ad/volumes" Oct 30 14:58:48.569415 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:48.569342 5824 generic.go:356] "Generic (PLEG): container finished" podID="9687edf9-7a77-4f59-83e6-b9e87e4ce4df" containerID="4745f99c433e1e62dc32a0c49eb71ddec271e25c75b79689e620f2628e827f60" exitCode=0 Oct 30 14:58:48.569753 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:48.569428 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" event={"ID":"9687edf9-7a77-4f59-83e6-b9e87e4ce4df","Type":"ContainerDied","Data":"4745f99c433e1e62dc32a0c49eb71ddec271e25c75b79689e620f2628e827f60"} Oct 30 14:58:49.727198 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.727156 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:49.849569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.849482 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-openshift-ca\") pod \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " Oct 30 14:58:49.849569 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.849527 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-kube-api-access-h9s27\" (UniqueName: \"kubernetes.io/projected/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-kube-api-access-h9s27\") pod \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " Oct 30 14:58:49.849815 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.849608 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-xks-ca\") pod \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\" (UID: \"9687edf9-7a77-4f59-83e6-b9e87e4ce4df\") " Oct 30 14:58:49.849877 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.849835 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-openshift-ca" pod "9687edf9-7a77-4f59-83e6-b9e87e4ce4df" (UID: "9687edf9-7a77-4f59-83e6-b9e87e4ce4df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:58:49.849992 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.849974 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:49.851639 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.851613 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-xks-ca" pod "9687edf9-7a77-4f59-83e6-b9e87e4ce4df" (UID: "9687edf9-7a77-4f59-83e6-b9e87e4ce4df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:58:49.851793 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.851774 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-kube-api-access-h9s27" pod "9687edf9-7a77-4f59-83e6-b9e87e4ce4df" (UID: "9687edf9-7a77-4f59-83e6-b9e87e4ce4df"). InnerVolumeSpecName "kube-api-access-h9s27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:58:49.950859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.950829 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:49.950859 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:49.950854 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9s27\" (UniqueName: \"kubernetes.io/projected/9687edf9-7a77-4f59-83e6-b9e87e4ce4df-kube-api-access-h9s27\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.194879 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.194852 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:58:50.253681 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.253655 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93e9207-0517-44a4-a0ec-0233acd50963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93e9207-0517-44a4-a0ec-0233acd50963-istiod-ca-cert\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.253852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.253701 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-kube-api-access-rnrwg\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-kube-api-access-rnrwg\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.253852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.253738 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-envoy\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.253852 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.253799 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-certs\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.254065 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254046 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-certs" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:58:50.254141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254065 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93e9207-0517-44a4-a0ec-0233acd50963-istiod-ca-cert" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:58:50.254141 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254117 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a93e9207-0517-44a4-a0ec-0233acd50963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93e9207-0517-44a4-a0ec-0233acd50963-istio-podinfo\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.254276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254163 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-socket\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.254276 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254220 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-data\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.254385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254298 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-credential-socket\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.254385 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254335 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-istio-token\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-istio-token\") pod \"a93e9207-0517-44a4-a0ec-0233acd50963\" (UID: \"a93e9207-0517-44a4-a0ec-0233acd50963\") " Oct 30 14:58:50.254905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254761 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-credential-socket" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:58:50.254905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254818 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.254905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254839 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93e9207-0517-44a4-a0ec-0233acd50963-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.254905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254834 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-data" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:58:50.254905 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.254874 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-socket" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:58:50.256494 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.256466 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-envoy" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:58:50.256599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.256509 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-kube-api-access-rnrwg" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "kube-api-access-rnrwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:58:50.256599 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.256525 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a93e9207-0517-44a4-a0ec-0233acd50963-istio-podinfo" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 14:58:50.256689 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.256642 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-istio-token" pod "a93e9207-0517-44a4-a0ec-0233acd50963" (UID: "a93e9207-0517-44a4-a0ec-0233acd50963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:58:50.356133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.356103 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.356133 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.356130 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.356329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.356141 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.356329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.356150 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.356329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.356159 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnrwg\" (UniqueName: \"kubernetes.io/projected/a93e9207-0517-44a4-a0ec-0233acd50963-kube-api-access-rnrwg\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.356329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.356167 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93e9207-0517-44a4-a0ec-0233acd50963-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.356329 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.356176 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93e9207-0517-44a4-a0ec-0233acd50963-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:58:50.580518 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.580492 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" Oct 30 14:58:50.580696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.580514 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2" event={"ID":"9687edf9-7a77-4f59-83e6-b9e87e4ce4df","Type":"ContainerDied","Data":"57eb2feb802b7c6a888fbd2d4094074eaa615f3928b2193202bc630b0bce3ac8"} Oct 30 14:58:50.580696 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.580571 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57eb2feb802b7c6a888fbd2d4094074eaa615f3928b2193202bc630b0bce3ac8" Oct 30 14:58:50.582706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.582476 5824 generic.go:356] "Generic (PLEG): container finished" podID="a93e9207-0517-44a4-a0ec-0233acd50963" containerID="2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9" exitCode=0 Oct 30 14:58:50.582706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.582526 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" event={"ID":"a93e9207-0517-44a4-a0ec-0233acd50963","Type":"ContainerDied","Data":"2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9"} Oct 30 14:58:50.582706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.582541 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" Oct 30 14:58:50.582706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.582571 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx" event={"ID":"a93e9207-0517-44a4-a0ec-0233acd50963","Type":"ContainerDied","Data":"d816f55a404ab2709718123bc39c22f1302aeec713537f3d6bffc9e8eeb1b577"} Oct 30 14:58:50.582706 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.582594 5824 scope.go:122] "RemoveContainer" containerID="2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9" Oct 30 14:58:50.591960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.591945 5824 scope.go:122] "RemoveContainer" containerID="b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca" Oct 30 14:58:50.599939 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.599913 5824 scope.go:122] "RemoveContainer" containerID="2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9" Oct 30 14:58:50.600315 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:58:50.600184 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9\": container with ID starting with 2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9 not found: ID does not exist" containerID="2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9" Oct 30 14:58:50.600315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.600231 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9"} err="failed to get container status \"2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9\": rpc error: code = NotFound desc = could not find container \"2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9\": container with ID starting with 2db40360a4841654c806428abc69dbc078caaaffc3ef95e715ec087d20f7cdc9 not found: ID does not exist" Oct 30 14:58:50.600315 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.600248 5824 scope.go:122] "RemoveContainer" containerID="b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca" Oct 30 14:58:50.600505 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:58:50.600454 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca\": container with ID starting with b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca not found: ID does not exist" containerID="b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca" Oct 30 14:58:50.600505 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.600469 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca"} err="failed to get container status \"b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca\": rpc error: code = NotFound desc = could not find container \"b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca\": container with ID starting with b357b9b0fb7dfaea7730d3d8cbd14ffc63eb928b993b5024ec19ebdbd0815aca not found: ID does not exist" Oct 30 14:58:50.641834 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.641797 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx"] Oct 30 14:58:50.655225 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:50.655182 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d3d611-openshift-default-55fb7bcb8f6z2tx"] Oct 30 14:58:52.315580 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:52.315546 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" path="/var/lib/kubelet/pods/a93e9207-0517-44a4-a0ec-0233acd50963/volumes" Oct 30 14:58:53.092140 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:53.092103 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" podUID="6955f413-b60c-4f55-b7f2-a72b9b497499" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:58:53.092319 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:53.092144 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" podUID="719c4136-7e7a-4d4d-92e0-c81510a6904e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 14:58:59.040099 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:59.040063 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn"] Oct 30 14:58:59.052589 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:58:59.052566 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-2vbjn"] Oct 30 14:59:00.315496 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:00.315461 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb8f498-7601-4fa2-99e9-48ac5faacbcb" path="/var/lib/kubelet/pods/beb8f498-7601-4fa2-99e9-48ac5faacbcb/volumes" Oct 30 14:59:01.310998 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.310971 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:59:01.338321 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.338301 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:59:01.447801 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.447775 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719c4136-7e7a-4d4d-92e0-c81510a6904e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719c4136-7e7a-4d4d-92e0-c81510a6904e-plugins-config-volume\") pod \"719c4136-7e7a-4d4d-92e0-c81510a6904e\" (UID: \"719c4136-7e7a-4d4d-92e0-c81510a6904e\") " Oct 30 14:59:01.447960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.447810 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6955f413-b60c-4f55-b7f2-a72b9b497499-kube-api-access-rqh98\" (UniqueName: \"kubernetes.io/projected/6955f413-b60c-4f55-b7f2-a72b9b497499-kube-api-access-rqh98\") pod \"6955f413-b60c-4f55-b7f2-a72b9b497499\" (UID: \"6955f413-b60c-4f55-b7f2-a72b9b497499\") " Oct 30 14:59:01.447960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.447830 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6955f413-b60c-4f55-b7f2-a72b9b497499-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6955f413-b60c-4f55-b7f2-a72b9b497499-plugins-config-volume\") pod \"6955f413-b60c-4f55-b7f2-a72b9b497499\" (UID: \"6955f413-b60c-4f55-b7f2-a72b9b497499\") " Oct 30 14:59:01.447960 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.447901 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719c4136-7e7a-4d4d-92e0-c81510a6904e-kube-api-access-whrl9\" (UniqueName: \"kubernetes.io/projected/719c4136-7e7a-4d4d-92e0-c81510a6904e-kube-api-access-whrl9\") pod \"719c4136-7e7a-4d4d-92e0-c81510a6904e\" (UID: \"719c4136-7e7a-4d4d-92e0-c81510a6904e\") " Oct 30 14:59:01.448245 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.448214 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719c4136-7e7a-4d4d-92e0-c81510a6904e-plugins-config-volume" pod "719c4136-7e7a-4d4d-92e0-c81510a6904e" (UID: "719c4136-7e7a-4d4d-92e0-c81510a6904e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:59:01.448341 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.448254 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6955f413-b60c-4f55-b7f2-a72b9b497499-plugins-config-volume" pod "6955f413-b60c-4f55-b7f2-a72b9b497499" (UID: "6955f413-b60c-4f55-b7f2-a72b9b497499"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:59:01.449957 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.449931 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719c4136-7e7a-4d4d-92e0-c81510a6904e-kube-api-access-whrl9" pod "719c4136-7e7a-4d4d-92e0-c81510a6904e" (UID: "719c4136-7e7a-4d4d-92e0-c81510a6904e"). InnerVolumeSpecName "kube-api-access-whrl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:59:01.450061 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.449965 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6955f413-b60c-4f55-b7f2-a72b9b497499-kube-api-access-rqh98" pod "6955f413-b60c-4f55-b7f2-a72b9b497499" (UID: "6955f413-b60c-4f55-b7f2-a72b9b497499"). InnerVolumeSpecName "kube-api-access-rqh98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:59:01.548756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.548706 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrl9\" (UniqueName: \"kubernetes.io/projected/719c4136-7e7a-4d4d-92e0-c81510a6904e-kube-api-access-whrl9\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:59:01.548756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.548728 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719c4136-7e7a-4d4d-92e0-c81510a6904e-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:59:01.548756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.548739 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqh98\" (UniqueName: \"kubernetes.io/projected/6955f413-b60c-4f55-b7f2-a72b9b497499-kube-api-access-rqh98\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:59:01.548756 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.548747 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6955f413-b60c-4f55-b7f2-a72b9b497499-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 14:59:01.642261 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.642237 5824 generic.go:356] "Generic (PLEG): container finished" podID="6955f413-b60c-4f55-b7f2-a72b9b497499" containerID="16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12" exitCode=0 Oct 30 14:59:01.642372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.642289 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" Oct 30 14:59:01.642372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.642312 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" event={"ID":"6955f413-b60c-4f55-b7f2-a72b9b497499","Type":"ContainerDied","Data":"16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12"} Oct 30 14:59:01.642372 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.642361 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886" event={"ID":"6955f413-b60c-4f55-b7f2-a72b9b497499","Type":"ContainerDied","Data":"0a1247c466c8bbefbf8b415050804be1e5c667ec192a89a8c5ef3fea12a63e99"} Oct 30 14:59:01.642583 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.642389 5824 scope.go:122] "RemoveContainer" containerID="16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12" Oct 30 14:59:01.643694 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.643674 5824 generic.go:356] "Generic (PLEG): container finished" podID="719c4136-7e7a-4d4d-92e0-c81510a6904e" containerID="7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499" exitCode=0 Oct 30 14:59:01.643788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.643714 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" Oct 30 14:59:01.643788 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.643763 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" event={"ID":"719c4136-7e7a-4d4d-92e0-c81510a6904e","Type":"ContainerDied","Data":"7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499"} Oct 30 14:59:01.643866 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.643820 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n" event={"ID":"719c4136-7e7a-4d4d-92e0-c81510a6904e","Type":"ContainerDied","Data":"4ebfb101ec0502fb54cfef5bd7c6bae74a27a8ce7a83c3b41e16316921530800"} Oct 30 14:59:01.652474 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.652453 5824 scope.go:122] "RemoveContainer" containerID="16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12" Oct 30 14:59:01.652758 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:59:01.652731 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12\": container with ID starting with 16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12 not found: ID does not exist" containerID="16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12" Oct 30 14:59:01.652858 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.652761 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12"} err="failed to get container status \"16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12\": rpc error: code = NotFound desc = could not find container \"16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12\": container with ID starting with 16f45121b8a623fa09082c4ec2d493170b184a8ee303c8a7b377133a723e6d12 not found: ID does not exist" Oct 30 14:59:01.652858 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.652786 5824 scope.go:122] "RemoveContainer" containerID="7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499" Oct 30 14:59:01.660824 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.660807 5824 scope.go:122] "RemoveContainer" containerID="7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499" Oct 30 14:59:01.661103 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:59:01.661080 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499\": container with ID starting with 7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499 not found: ID does not exist" containerID="7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499" Oct 30 14:59:01.661182 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.661113 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499"} err="failed to get container status \"7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499\": rpc error: code = NotFound desc = could not find container \"7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499\": container with ID starting with 7033ec3c2f016d56e40032ac6b3a55ae608886f484444635631462ba421d8499 not found: ID does not exist" Oct 30 14:59:01.700432 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.700409 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886"] Oct 30 14:59:01.706275 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.706244 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d3d611-8589btn886"] Oct 30 14:59:01.745962 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.745935 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n"] Oct 30 14:59:01.751436 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:01.751416 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d3d611-667f65b9bwtw5n"] Oct 30 14:59:02.315766 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:02.315689 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6955f413-b60c-4f55-b7f2-a72b9b497499" path="/var/lib/kubelet/pods/6955f413-b60c-4f55-b7f2-a72b9b497499/volumes" Oct 30 14:59:02.316078 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:02.316063 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719c4136-7e7a-4d4d-92e0-c81510a6904e" path="/var/lib/kubelet/pods/719c4136-7e7a-4d4d-92e0-c81510a6904e/volumes" Oct 30 14:59:03.640322 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:59:03.640281 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:59:31.054024 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:31.053992 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp"] Oct 30 14:59:31.066952 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:31.066924 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8h8dp"] Oct 30 14:59:32.315839 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:32.315802 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f61cb3-3582-40a0-8384-1ecd709a115f" path="/var/lib/kubelet/pods/c4f61cb3-3582-40a0-8384-1ecd709a115f/volumes" Oct 30 14:59:33.281662 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:33.281630 5824 scope.go:122] "RemoveContainer" containerID="27c17a43cb0f2067f6156d443feab211084082ca098860b65aebd8839cd5da0b" Oct 30 14:59:33.292614 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:33.292586 5824 scope.go:122] "RemoveContainer" containerID="5faed051d9283f355c7b8562284f20f47000977720828344e80c6d7268e17a0d" Oct 30 14:59:33.300913 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:33.300887 5824 scope.go:122] "RemoveContainer" containerID="6ecfc231dc57ac39d39d2d5fe263962416c71b59c4c6e20b6c3c2b87529646d0" Oct 30 14:59:33.640614 ip-10-0-131-200 kubenswrapper[5824]: E1030 14:59:33.640529 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 14:59:59.068075 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:59.068041 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg"] Oct 30 14:59:59.083728 ip-10-0-131-200 kubenswrapper[5824]: I1030 14:59:59.083702 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-46wmg"] Oct 30 15:00:00.172851 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.172818 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k"] Oct 30 15:00:00.173342 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173323 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" Oct 30 15:00:00.173401 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173344 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" Oct 30 15:00:00.173401 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173366 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" Oct 30 15:00:00.173401 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173375 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" Oct 30 15:00:00.173401 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173385 5824 cpu_manager.go:414] "containerMap: removing container" podUID="6955f413-b60c-4f55-b7f2-a72b9b497499" containerName="payload-pre-processing" Oct 30 15:00:00.173401 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173391 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6955f413-b60c-4f55-b7f2-a72b9b497499" containerName="payload-pre-processing" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173404 5824 cpu_manager.go:414] "containerMap: removing container" podUID="719c4136-7e7a-4d4d-92e0-c81510a6904e" containerName="payload-processing" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173409 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719c4136-7e7a-4d4d-92e0-c81510a6904e" containerName="payload-processing" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173429 5824 cpu_manager.go:414] "containerMap: removing container" podUID="9687edf9-7a77-4f59-83e6-b9e87e4ce4df" containerName="revoke-keys" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173435 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9687edf9-7a77-4f59-83e6-b9e87e4ce4df" containerName="revoke-keys" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173504 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173516 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93e9207-0517-44a4-a0ec-0233acd50963" containerName="istio-proxy" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173526 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="6955f413-b60c-4f55-b7f2-a72b9b497499" containerName="payload-pre-processing" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173536 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="719c4136-7e7a-4d4d-92e0-c81510a6904e" containerName="payload-processing" Oct 30 15:00:00.173567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.173544 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="9687edf9-7a77-4f59-83e6-b9e87e4ce4df" containerName="revoke-keys" Oct 30 15:00:00.176702 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.176686 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:00.194584 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.194558 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k"] Oct 30 15:00:00.198733 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.198708 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de84d9bc-8211-4c3c-8d08-80fae7005301-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889540-bm79k\" (UID: \"de84d9bc-8211-4c3c-8d08-80fae7005301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:00.198854 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.198750 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bttb\" (UniqueName: \"kubernetes.io/projected/de84d9bc-8211-4c3c-8d08-80fae7005301-kube-api-access-6bttb\") pod \"maas-api-key-cleanup-29889540-bm79k\" (UID: \"de84d9bc-8211-4c3c-8d08-80fae7005301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:00.206208 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.206164 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:00:00.207404 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.207382 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:00:00.207485 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.207414 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:00:00.299823 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.299782 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de84d9bc-8211-4c3c-8d08-80fae7005301-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889540-bm79k\" (UID: \"de84d9bc-8211-4c3c-8d08-80fae7005301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:00.300011 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.299828 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bttb\" (UniqueName: \"kubernetes.io/projected/de84d9bc-8211-4c3c-8d08-80fae7005301-kube-api-access-6bttb\") pod \"maas-api-key-cleanup-29889540-bm79k\" (UID: \"de84d9bc-8211-4c3c-8d08-80fae7005301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:00.300428 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.300409 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de84d9bc-8211-4c3c-8d08-80fae7005301-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889540-bm79k\" (UID: \"de84d9bc-8211-4c3c-8d08-80fae7005301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:00.315989 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.315960 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f817cf-e772-44d7-a474-3a2ffc8c4368" path="/var/lib/kubelet/pods/75f817cf-e772-44d7-a474-3a2ffc8c4368/volumes" Oct 30 15:00:00.328939 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.328918 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bttb\" (UniqueName: \"kubernetes.io/projected/de84d9bc-8211-4c3c-8d08-80fae7005301-kube-api-access-6bttb\") pod \"maas-api-key-cleanup-29889540-bm79k\" (UID: \"de84d9bc-8211-4c3c-8d08-80fae7005301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:00.488915 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.488885 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:00.794175 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.794142 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k"] Oct 30 15:00:00.794587 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:00:00.794560 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde84d9bc_8211_4c3c_8d08_80fae7005301.slice/crio-aec1c24a5a2e673c9a97c7495e5eb55f03447b747ba850feb7bfd9f4f7cc06d4 WatchSource:0}: Error finding container aec1c24a5a2e673c9a97c7495e5eb55f03447b747ba850feb7bfd9f4f7cc06d4: Status 404 returned error can't find the container with id aec1c24a5a2e673c9a97c7495e5eb55f03447b747ba850feb7bfd9f4f7cc06d4 Oct 30 15:00:00.969453 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:00.969421 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" event={"ID":"de84d9bc-8211-4c3c-8d08-80fae7005301","Type":"ContainerStarted","Data":"aec1c24a5a2e673c9a97c7495e5eb55f03447b747ba850feb7bfd9f4f7cc06d4"} Oct 30 15:00:01.976543 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:01.976505 5824 generic.go:356] "Generic (PLEG): container finished" podID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerID="b54a17357b1a272f6465b7396e478d1610b0526a217ef577dcc7cf90d2d76e73" exitCode=60 Oct 30 15:00:01.976934 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:01.976559 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" event={"ID":"de84d9bc-8211-4c3c-8d08-80fae7005301","Type":"ContainerDied","Data":"b54a17357b1a272f6465b7396e478d1610b0526a217ef577dcc7cf90d2d76e73"} Oct 30 15:00:01.976934 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:01.976892 5824 scope.go:122] "RemoveContainer" containerID="b54a17357b1a272f6465b7396e478d1610b0526a217ef577dcc7cf90d2d76e73" Oct 30 15:00:02.984069 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:02.984028 5824 generic.go:356] "Generic (PLEG): container finished" podID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerID="4bfde87d7734e0656e6c400f8b5d82334432fda50c923d083ddd7e39a4112b3d" exitCode=60 Oct 30 15:00:02.984600 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:02.984118 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" event={"ID":"de84d9bc-8211-4c3c-8d08-80fae7005301","Type":"ContainerDied","Data":"4bfde87d7734e0656e6c400f8b5d82334432fda50c923d083ddd7e39a4112b3d"} Oct 30 15:00:02.984600 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:02.984214 5824 scope.go:122] "RemoveContainer" containerID="b54a17357b1a272f6465b7396e478d1610b0526a217ef577dcc7cf90d2d76e73" Oct 30 15:00:02.984600 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:02.984487 5824 scope.go:122] "RemoveContainer" containerID="4bfde87d7734e0656e6c400f8b5d82334432fda50c923d083ddd7e39a4112b3d" Oct 30 15:00:02.984789 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:00:02.984770 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889540-bm79k_odh-ai-gateway-infra(de84d9bc-8211-4c3c-8d08-80fae7005301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" Oct 30 15:00:03.640958 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:00:03.640922 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:00:03.990872 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:03.990848 5824 scope.go:122] "RemoveContainer" containerID="4bfde87d7734e0656e6c400f8b5d82334432fda50c923d083ddd7e39a4112b3d" Oct 30 15:00:03.991304 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:00:03.991053 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889540-bm79k_odh-ai-gateway-infra(de84d9bc-8211-4c3c-8d08-80fae7005301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" Oct 30 15:00:14.319912 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:14.319834 5824 scope.go:122] "RemoveContainer" containerID="4bfde87d7734e0656e6c400f8b5d82334432fda50c923d083ddd7e39a4112b3d" Oct 30 15:00:15.046837 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:15.046799 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" event={"ID":"de84d9bc-8211-4c3c-8d08-80fae7005301","Type":"ContainerStarted","Data":"8d5b68d158cb8839b1f13823c7b676fb5f55690a94b09c7ff443ca4a01297432"} Oct 30 15:00:15.094845 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:15.094794 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" podStartSLOduration=15.094780687 podStartE2EDuration="15.094780687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:15.093686318 +0000 UTC m=+4244.894191761" watchObservedRunningTime="2026-10-30 15:00:15.094780687 +0000 UTC m=+4244.895286186" Oct 30 15:00:16.054062 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:16.054026 5824 generic.go:356] "Generic (PLEG): container finished" podID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerID="8d5b68d158cb8839b1f13823c7b676fb5f55690a94b09c7ff443ca4a01297432" exitCode=60 Oct 30 15:00:16.054521 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:16.054098 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" event={"ID":"de84d9bc-8211-4c3c-8d08-80fae7005301","Type":"ContainerDied","Data":"8d5b68d158cb8839b1f13823c7b676fb5f55690a94b09c7ff443ca4a01297432"} Oct 30 15:00:16.054521 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:16.054157 5824 scope.go:122] "RemoveContainer" containerID="4bfde87d7734e0656e6c400f8b5d82334432fda50c923d083ddd7e39a4112b3d" Oct 30 15:00:16.054521 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:16.054436 5824 scope.go:122] "RemoveContainer" containerID="8d5b68d158cb8839b1f13823c7b676fb5f55690a94b09c7ff443ca4a01297432" Oct 30 15:00:16.054658 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:00:16.054636 5824 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889540-bm79k_odh-ai-gateway-infra(de84d9bc-8211-4c3c-8d08-80fae7005301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" Oct 30 15:00:16.106114 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:16.106087 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k"] Oct 30 15:00:17.181595 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:17.181573 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:17.259978 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:17.259944 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de84d9bc-8211-4c3c-8d08-80fae7005301-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de84d9bc-8211-4c3c-8d08-80fae7005301-maas-api-service-ca\") pod \"de84d9bc-8211-4c3c-8d08-80fae7005301\" (UID: \"de84d9bc-8211-4c3c-8d08-80fae7005301\") " Oct 30 15:00:17.260200 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:17.260100 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de84d9bc-8211-4c3c-8d08-80fae7005301-kube-api-access-6bttb\" (UniqueName: \"kubernetes.io/projected/de84d9bc-8211-4c3c-8d08-80fae7005301-kube-api-access-6bttb\") pod \"de84d9bc-8211-4c3c-8d08-80fae7005301\" (UID: \"de84d9bc-8211-4c3c-8d08-80fae7005301\") " Oct 30 15:00:17.260337 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:17.260312 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de84d9bc-8211-4c3c-8d08-80fae7005301-maas-api-service-ca" pod "de84d9bc-8211-4c3c-8d08-80fae7005301" (UID: "de84d9bc-8211-4c3c-8d08-80fae7005301"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:17.260472 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:17.260459 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de84d9bc-8211-4c3c-8d08-80fae7005301-maas-api-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:00:17.262456 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:17.262429 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de84d9bc-8211-4c3c-8d08-80fae7005301-kube-api-access-6bttb" pod "de84d9bc-8211-4c3c-8d08-80fae7005301" (UID: "de84d9bc-8211-4c3c-8d08-80fae7005301"). InnerVolumeSpecName "kube-api-access-6bttb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:17.361755 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:17.361679 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bttb\" (UniqueName: \"kubernetes.io/projected/de84d9bc-8211-4c3c-8d08-80fae7005301-kube-api-access-6bttb\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:00:18.066241 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:18.066204 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" event={"ID":"de84d9bc-8211-4c3c-8d08-80fae7005301","Type":"ContainerDied","Data":"aec1c24a5a2e673c9a97c7495e5eb55f03447b747ba850feb7bfd9f4f7cc06d4"} Oct 30 15:00:18.066462 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:18.066245 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k" Oct 30 15:00:18.066462 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:18.066269 5824 scope.go:122] "RemoveContainer" containerID="8d5b68d158cb8839b1f13823c7b676fb5f55690a94b09c7ff443ca4a01297432" Oct 30 15:00:18.126696 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:18.126669 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k"] Oct 30 15:00:18.140097 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:18.140070 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-bm79k"] Oct 30 15:00:18.315662 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:18.315630 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" path="/var/lib/kubelet/pods/de84d9bc-8211-4c3c-8d08-80fae7005301/volumes" Oct 30 15:00:29.044072 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:29.044030 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz"] Oct 30 15:00:29.060858 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:29.060822 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ddssz"] Oct 30 15:00:30.316156 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:30.316113 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20b4442-c4f3-4c68-8974-c497b27fc517" path="/var/lib/kubelet/pods/b20b4442-c4f3-4c68-8974-c497b27fc517/volumes" Oct 30 15:00:33.339846 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:33.339812 5824 scope.go:122] "RemoveContainer" containerID="fe907441b7cf05639dce956e30397bb0a1683e52106f0f50ed36fcacdc4c2839" Oct 30 15:00:33.348781 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:33.348760 5824 scope.go:122] "RemoveContainer" containerID="618a8cded7990162b3b0af5fbd594934bad29919f316b10c860fdf9234ffea4f" Oct 30 15:00:33.356477 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:33.356450 5824 scope.go:122] "RemoveContainer" containerID="48c1e667eacc61b55ac7dc0370d23fac2106be1ec81806f0f02c02e45a8e76bf" Oct 30 15:00:33.644252 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:00:33.644151 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:00:46.035849 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.035808 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg"] Oct 30 15:00:46.049096 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.049065 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qpjwg"] Oct 30 15:00:46.159008 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.158980 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs"] Oct 30 15:00:46.159401 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.159387 5824 cpu_manager.go:414] "containerMap: removing container" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:00:46.159458 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.159402 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:00:46.159458 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.159414 5824 cpu_manager.go:414] "containerMap: removing container" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:00:46.159458 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.159419 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:00:46.159568 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.159497 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:00:46.159568 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.159510 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:00:46.159568 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.159516 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:00:46.162464 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.162447 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.181470 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.181442 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:00:46.181611 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.181558 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:00:46.181675 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.181645 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:00:46.182391 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.182365 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs"] Oct 30 15:00:46.193053 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.193034 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:00:46.273177 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.273150 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtqz\" (UniqueName: \"kubernetes.io/projected/30542cd7-a500-44d0-8fd1-6cbc316b9d28-kube-api-access-xrtqz\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.273334 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.273320 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.273389 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.273353 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.315011 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.314952 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fce53e3-5e81-4e21-89f3-f3e3d87e92d8" path="/var/lib/kubelet/pods/1fce53e3-5e81-4e21-89f3-f3e3d87e92d8/volumes" Oct 30 15:00:46.374306 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.374279 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.374403 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.374316 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.374463 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.374434 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtqz\" (UniqueName: \"kubernetes.io/projected/30542cd7-a500-44d0-8fd1-6cbc316b9d28-kube-api-access-xrtqz\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.374869 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.374852 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.376680 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.376661 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.403643 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.403620 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtqz\" (UniqueName: \"kubernetes.io/projected/30542cd7-a500-44d0-8fd1-6cbc316b9d28-kube-api-access-xrtqz\") pod \"maas-api-revoke-sub-d576e68b-qwrcs\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.474919 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.474901 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:46.772404 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:46.772377 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs"] Oct 30 15:00:46.773161 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:00:46.773137 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30542cd7_a500_44d0_8fd1_6cbc316b9d28.slice/crio-0c38edb095552b96fa794681b22f883566101edc4a9c85d1514b09f98bce64bc WatchSource:0}: Error finding container 0c38edb095552b96fa794681b22f883566101edc4a9c85d1514b09f98bce64bc: Status 404 returned error can't find the container with id 0c38edb095552b96fa794681b22f883566101edc4a9c85d1514b09f98bce64bc Oct 30 15:00:47.220566 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:47.220528 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" event={"ID":"30542cd7-a500-44d0-8fd1-6cbc316b9d28","Type":"ContainerStarted","Data":"50664dccfafaca5ce14ae5503bef7ba7c4de7cafc5804ad5f1e88f8f2ec95f8a"} Oct 30 15:00:47.221002 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:47.220590 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" event={"ID":"30542cd7-a500-44d0-8fd1-6cbc316b9d28","Type":"ContainerStarted","Data":"0c38edb095552b96fa794681b22f883566101edc4a9c85d1514b09f98bce64bc"} Oct 30 15:00:47.266197 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:47.266137 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" podStartSLOduration=1.266125999 podStartE2EDuration="1.266125999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:47.264959881 +0000 UTC m=+4277.065465319" watchObservedRunningTime="2026-10-30 15:00:47.266125999 +0000 UTC m=+4277.066631444" Oct 30 15:00:48.233615 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:48.233576 5824 generic.go:356] "Generic (PLEG): container finished" podID="30542cd7-a500-44d0-8fd1-6cbc316b9d28" containerID="50664dccfafaca5ce14ae5503bef7ba7c4de7cafc5804ad5f1e88f8f2ec95f8a" exitCode=0 Oct 30 15:00:48.234070 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:48.233656 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" event={"ID":"30542cd7-a500-44d0-8fd1-6cbc316b9d28","Type":"ContainerDied","Data":"50664dccfafaca5ce14ae5503bef7ba7c4de7cafc5804ad5f1e88f8f2ec95f8a"} Oct 30 15:00:49.371155 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.371130 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:49.507324 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.507257 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-xks-ca\") pod \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " Oct 30 15:00:49.507324 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.507314 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-openshift-ca\") pod \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " Oct 30 15:00:49.507510 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.507385 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30542cd7-a500-44d0-8fd1-6cbc316b9d28-kube-api-access-xrtqz\" (UniqueName: \"kubernetes.io/projected/30542cd7-a500-44d0-8fd1-6cbc316b9d28-kube-api-access-xrtqz\") pod \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\" (UID: \"30542cd7-a500-44d0-8fd1-6cbc316b9d28\") " Oct 30 15:00:49.507730 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.507705 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-openshift-ca" pod "30542cd7-a500-44d0-8fd1-6cbc316b9d28" (UID: "30542cd7-a500-44d0-8fd1-6cbc316b9d28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:49.509342 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.509316 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-xks-ca" pod "30542cd7-a500-44d0-8fd1-6cbc316b9d28" (UID: "30542cd7-a500-44d0-8fd1-6cbc316b9d28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:00:49.509442 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.509423 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30542cd7-a500-44d0-8fd1-6cbc316b9d28-kube-api-access-xrtqz" pod "30542cd7-a500-44d0-8fd1-6cbc316b9d28" (UID: "30542cd7-a500-44d0-8fd1-6cbc316b9d28"). InnerVolumeSpecName "kube-api-access-xrtqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:49.608013 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.607985 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrtqz\" (UniqueName: \"kubernetes.io/projected/30542cd7-a500-44d0-8fd1-6cbc316b9d28-kube-api-access-xrtqz\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:00:49.608013 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.608011 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:00:49.608198 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:49.608021 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30542cd7-a500-44d0-8fd1-6cbc316b9d28-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:00:50.244849 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:50.244818 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" Oct 30 15:00:50.245046 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:50.244843 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs" event={"ID":"30542cd7-a500-44d0-8fd1-6cbc316b9d28","Type":"ContainerDied","Data":"0c38edb095552b96fa794681b22f883566101edc4a9c85d1514b09f98bce64bc"} Oct 30 15:00:50.245046 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:00:50.244920 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c38edb095552b96fa794681b22f883566101edc4a9c85d1514b09f98bce64bc" Oct 30 15:01:01.364214 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.363916 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz"] Oct 30 15:01:01.364853 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.364560 5824 cpu_manager.go:414] "containerMap: removing container" podUID="30542cd7-a500-44d0-8fd1-6cbc316b9d28" containerName="revoke-keys" Oct 30 15:01:01.364853 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.364581 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30542cd7-a500-44d0-8fd1-6cbc316b9d28" containerName="revoke-keys" Oct 30 15:01:01.364853 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.364610 5824 cpu_manager.go:414] "containerMap: removing container" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:01:01.364853 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.364619 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de84d9bc-8211-4c3c-8d08-80fae7005301" containerName="cleanup" Oct 30 15:01:01.364853 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.364730 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="30542cd7-a500-44d0-8fd1-6cbc316b9d28" containerName="revoke-keys" Oct 30 15:01:01.372540 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.372516 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.407214 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.385263 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz"] Oct 30 15:01:01.407880 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.400391 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:01.408064 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.400505 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:01.408215 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.400619 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:01.429272 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.427174 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:01:01.535618 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.535587 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.535791 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.535644 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.535791 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.535734 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxcn\" (UniqueName: \"kubernetes.io/projected/02600861-c39c-4219-95fc-c2ca52e0bd43-kube-api-access-jnxcn\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.636859 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.636783 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.636859 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.636844 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.637060 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.636937 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxcn\" (UniqueName: \"kubernetes.io/projected/02600861-c39c-4219-95fc-c2ca52e0bd43-kube-api-access-jnxcn\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.637629 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.637602 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.639056 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.639026 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.703212 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.700207 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxcn\" (UniqueName: \"kubernetes.io/projected/02600861-c39c-4219-95fc-c2ca52e0bd43-kube-api-access-jnxcn\") pod \"maas-api-revoke-sub-d2bc93bc-jl9mz\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:01.737653 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:01.737618 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:02.195262 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:02.182671 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz"] Oct 30 15:01:02.332332 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:02.332293 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" event={"ID":"02600861-c39c-4219-95fc-c2ca52e0bd43","Type":"ContainerStarted","Data":"fddeef52a91075534dfc935ace395db17c419988c7e6dc027b8c68ffb85ba40a"} Oct 30 15:01:03.335932 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:03.335902 5824 generic.go:356] "Generic (PLEG): container finished" podID="02600861-c39c-4219-95fc-c2ca52e0bd43" containerID="a1c89379351ee925f5e98f9327d70518c7cc65b0a1c38e92606fd94c44b40961" exitCode=0 Oct 30 15:01:03.336289 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:03.335962 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" event={"ID":"02600861-c39c-4219-95fc-c2ca52e0bd43","Type":"ContainerDied","Data":"a1c89379351ee925f5e98f9327d70518c7cc65b0a1c38e92606fd94c44b40961"} Oct 30 15:01:03.642140 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:01:03.642056 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:01:04.480091 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.480064 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:04.611084 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.611006 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-xks-ca\") pod \"02600861-c39c-4219-95fc-c2ca52e0bd43\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " Oct 30 15:01:04.611272 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.611091 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02600861-c39c-4219-95fc-c2ca52e0bd43-kube-api-access-jnxcn\" (UniqueName: \"kubernetes.io/projected/02600861-c39c-4219-95fc-c2ca52e0bd43-kube-api-access-jnxcn\") pod \"02600861-c39c-4219-95fc-c2ca52e0bd43\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " Oct 30 15:01:04.611272 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.611143 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-openshift-ca\") pod \"02600861-c39c-4219-95fc-c2ca52e0bd43\" (UID: \"02600861-c39c-4219-95fc-c2ca52e0bd43\") " Oct 30 15:01:04.611550 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.611525 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-openshift-ca" pod "02600861-c39c-4219-95fc-c2ca52e0bd43" (UID: "02600861-c39c-4219-95fc-c2ca52e0bd43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:04.613036 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.613010 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-xks-ca" pod "02600861-c39c-4219-95fc-c2ca52e0bd43" (UID: "02600861-c39c-4219-95fc-c2ca52e0bd43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:04.613159 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.613142 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02600861-c39c-4219-95fc-c2ca52e0bd43-kube-api-access-jnxcn" pod "02600861-c39c-4219-95fc-c2ca52e0bd43" (UID: "02600861-c39c-4219-95fc-c2ca52e0bd43"). InnerVolumeSpecName "kube-api-access-jnxcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:04.712879 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.712856 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:01:04.712879 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.712879 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnxcn\" (UniqueName: \"kubernetes.io/projected/02600861-c39c-4219-95fc-c2ca52e0bd43-kube-api-access-jnxcn\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:01:04.713030 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:04.712890 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02600861-c39c-4219-95fc-c2ca52e0bd43-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:01:05.346548 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:05.346519 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" Oct 30 15:01:05.346548 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:05.346525 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz" event={"ID":"02600861-c39c-4219-95fc-c2ca52e0bd43","Type":"ContainerDied","Data":"fddeef52a91075534dfc935ace395db17c419988c7e6dc027b8c68ffb85ba40a"} Oct 30 15:01:05.346760 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:05.346567 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fddeef52a91075534dfc935ace395db17c419988c7e6dc027b8c68ffb85ba40a" Oct 30 15:01:13.551767 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.551710 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2"] Oct 30 15:01:13.552257 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.552237 5824 cpu_manager.go:414] "containerMap: removing container" podUID="02600861-c39c-4219-95fc-c2ca52e0bd43" containerName="revoke-keys" Oct 30 15:01:13.552318 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.552259 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02600861-c39c-4219-95fc-c2ca52e0bd43" containerName="revoke-keys" Oct 30 15:01:13.552384 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.552371 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="02600861-c39c-4219-95fc-c2ca52e0bd43" containerName="revoke-keys" Oct 30 15:01:13.555670 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.555646 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.586164 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.586134 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2ef0c7-openshift-default-dockercfg-wqwhz\"" Oct 30 15:01:13.588403 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.588375 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2"] Oct 30 15:01:13.594948 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.594920 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-token\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.595064 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.594959 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d28589-20c5-42e1-85eb-80a26e4df08d-istiod-ca-cert\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.595064 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.594980 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-data\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.595168 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.595089 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-envoy\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.595168 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.595117 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v68r\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-kube-api-access-6v68r\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.595168 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.595151 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-podinfo\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.595300 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.595257 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-certs\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.595300 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.595297 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-credential-socket\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.595383 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.595315 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-socket\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696441 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696410 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-socket\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696621 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696463 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-token\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696621 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696497 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d28589-20c5-42e1-85eb-80a26e4df08d-istiod-ca-cert\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696621 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696516 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-data\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696621 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696543 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-envoy\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696621 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696577 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v68r\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-kube-api-access-6v68r\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696940 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696642 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-podinfo\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696940 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696692 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-certs\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696940 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696757 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-credential-socket\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.696940 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.696875 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-socket\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.697096 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.697065 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-data\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.697096 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.697074 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-credential-socket\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.697337 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.697315 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-certs\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.697457 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.697313 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d28589-20c5-42e1-85eb-80a26e4df08d-istiod-ca-cert\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.698981 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.698962 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-envoy\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.699232 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.699211 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-podinfo\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.722523 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.722499 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-token\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.722929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.722910 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v68r\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-kube-api-access-6v68r\") pod \"e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:13.871020 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:13.870936 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:14.193087 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:14.193055 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2"] Oct 30 15:01:14.194956 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:01:14.194923 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d28589_20c5_42e1_85eb_80a26e4df08d.slice/crio-539d6675ac7735a2589d944dc97a60ebbc2d3bb6b2b05bf8d299ee5b1e0d195b WatchSource:0}: Error finding container 539d6675ac7735a2589d944dc97a60ebbc2d3bb6b2b05bf8d299ee5b1e0d195b: Status 404 returned error can't find the container with id 539d6675ac7735a2589d944dc97a60ebbc2d3bb6b2b05bf8d299ee5b1e0d195b Oct 30 15:01:14.197331 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:14.197290 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:01:14.197415 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:14.197379 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:01:14.197469 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:14.197429 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:01:14.392639 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:14.392592 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" event={"ID":"c3d28589-20c5-42e1-85eb-80a26e4df08d","Type":"ContainerStarted","Data":"d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5"} Oct 30 15:01:14.392855 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:14.392664 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" event={"ID":"c3d28589-20c5-42e1-85eb-80a26e4df08d","Type":"ContainerStarted","Data":"539d6675ac7735a2589d944dc97a60ebbc2d3bb6b2b05bf8d299ee5b1e0d195b"} Oct 30 15:01:14.444182 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:14.444070 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" podStartSLOduration=1.444054558 podStartE2EDuration="1.444054558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:14.442611583 +0000 UTC m=+4304.243117029" watchObservedRunningTime="2026-10-30 15:01:14.444054558 +0000 UTC m=+4304.244559999" Oct 30 15:01:14.871500 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:14.871423 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:15.876067 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:15.876035 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:16.404077 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:16.404046 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:16.405089 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:16.405067 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:28.702607 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.702565 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d"] Oct 30 15:01:28.707870 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.707845 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.728538 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.728500 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d"] Oct 30 15:01:28.739751 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.739718 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:28.739918 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.739758 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2ef0c7-serving-cert\"" Oct 30 15:01:28.739918 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.739764 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:28.739918 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.739817 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 15:01:28.741885 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.741868 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-54whp\"" Oct 30 15:01:28.841140 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.841111 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58l5\" (UniqueName: \"kubernetes.io/projected/4572cf7a-17e3-4448-b265-143028f333ab-kube-api-access-q58l5\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.841328 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.841143 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.841328 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.841168 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4572cf7a-17e3-4448-b265-143028f333ab-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.841328 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.841256 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.926845 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.926816 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j"] Oct 30 15:01:28.931490 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.931470 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:28.942135 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.942111 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58l5\" (UniqueName: \"kubernetes.io/projected/4572cf7a-17e3-4448-b265-143028f333ab-kube-api-access-q58l5\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.942272 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.942141 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.942272 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.942162 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4572cf7a-17e3-4448-b265-143028f333ab-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.942272 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.942178 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.942762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.942732 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4572cf7a-17e3-4448-b265-143028f333ab-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.944660 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.944640 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.944749 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.944689 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:28.949808 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.949787 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-70e4b7a8-dockercfg-7s7qp\"" Oct 30 15:01:28.950092 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.950073 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j"] Oct 30 15:01:28.950142 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.950119 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2ef0c7\"" Oct 30 15:01:28.970212 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:28.970137 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58l5\" (UniqueName: \"kubernetes.io/projected/4572cf7a-17e3-4448-b265-143028f333ab-kube-api-access-q58l5\") pod \"maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:29.025263 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.023867 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:29.042976 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.042942 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f14ee3-e72a-49fe-97ee-769c14dd0824-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j\" (UID: \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:29.043125 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.043097 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqh9b\" (UniqueName: \"kubernetes.io/projected/b7f14ee3-e72a-49fe-97ee-769c14dd0824-kube-api-access-wqh9b\") pod \"payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j\" (UID: \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:29.050465 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.050436 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm"] Oct 30 15:01:29.055058 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.055007 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:29.079823 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.079794 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm"] Oct 30 15:01:29.144640 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.144611 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f14ee3-e72a-49fe-97ee-769c14dd0824-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j\" (UID: \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:29.144803 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.144657 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkm56\" (UniqueName: \"kubernetes.io/projected/a801d113-37ee-4dbb-b972-9f93b36d6bd5-kube-api-access-zkm56\") pod \"payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm\" (UID: \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:29.144803 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.144738 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqh9b\" (UniqueName: \"kubernetes.io/projected/b7f14ee3-e72a-49fe-97ee-769c14dd0824-kube-api-access-wqh9b\") pod \"payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j\" (UID: \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:29.144803 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.144786 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a801d113-37ee-4dbb-b972-9f93b36d6bd5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm\" (UID: \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:29.146291 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.145481 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f14ee3-e72a-49fe-97ee-769c14dd0824-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j\" (UID: \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:29.171620 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.171596 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqh9b\" (UniqueName: \"kubernetes.io/projected/b7f14ee3-e72a-49fe-97ee-769c14dd0824-kube-api-access-wqh9b\") pod \"payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j\" (UID: \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:29.244929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.244877 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:29.246265 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.246239 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkm56\" (UniqueName: \"kubernetes.io/projected/a801d113-37ee-4dbb-b972-9f93b36d6bd5-kube-api-access-zkm56\") pod \"payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm\" (UID: \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:29.246334 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.246322 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a801d113-37ee-4dbb-b972-9f93b36d6bd5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm\" (UID: \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:29.246933 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.246916 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a801d113-37ee-4dbb-b972-9f93b36d6bd5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm\" (UID: \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:29.272585 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.272565 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkm56\" (UniqueName: \"kubernetes.io/projected/a801d113-37ee-4dbb-b972-9f93b36d6bd5-kube-api-access-zkm56\") pod \"payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm\" (UID: \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:29.340607 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.340570 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d"] Oct 30 15:01:29.341016 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:01:29.340972 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4572cf7a_17e3_4448_b265_143028f333ab.slice/crio-282bdf991dd8aadd9b4ef4bada3aaec7d47461c9bf3f2467f0ae59295c5f50aa WatchSource:0}: Error finding container 282bdf991dd8aadd9b4ef4bada3aaec7d47461c9bf3f2467f0ae59295c5f50aa: Status 404 returned error can't find the container with id 282bdf991dd8aadd9b4ef4bada3aaec7d47461c9bf3f2467f0ae59295c5f50aa Oct 30 15:01:29.342445 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.342426 5824 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:01:29.371243 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.371222 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:29.473117 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.473086 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" event={"ID":"4572cf7a-17e3-4448-b265-143028f333ab","Type":"ContainerStarted","Data":"282bdf991dd8aadd9b4ef4bada3aaec7d47461c9bf3f2467f0ae59295c5f50aa"} Oct 30 15:01:29.544292 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.544266 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j"] Oct 30 15:01:29.544815 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:01:29.544792 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f14ee3_e72a_49fe_97ee_769c14dd0824.slice/crio-2c81dcab9f3a0e73b1a8ea9b5a838f89fec3b7238576d4d0991035d2a23d2461 WatchSource:0}: Error finding container 2c81dcab9f3a0e73b1a8ea9b5a838f89fec3b7238576d4d0991035d2a23d2461: Status 404 returned error can't find the container with id 2c81dcab9f3a0e73b1a8ea9b5a838f89fec3b7238576d4d0991035d2a23d2461 Oct 30 15:01:29.673159 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:01:29.673125 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda801d113_37ee_4dbb_b972_9f93b36d6bd5.slice/crio-d82af5b1def6630942c15bc70284abe40d0b057545b70427202ad5b77727a8ef WatchSource:0}: Error finding container d82af5b1def6630942c15bc70284abe40d0b057545b70427202ad5b77727a8ef: Status 404 returned error can't find the container with id d82af5b1def6630942c15bc70284abe40d0b057545b70427202ad5b77727a8ef Oct 30 15:01:29.673308 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:29.673252 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm"] Oct 30 15:01:30.480291 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.480118 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" event={"ID":"4572cf7a-17e3-4448-b265-143028f333ab","Type":"ContainerStarted","Data":"b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18"} Oct 30 15:01:30.480921 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.480893 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:30.482715 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.482670 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" event={"ID":"b7f14ee3-e72a-49fe-97ee-769c14dd0824","Type":"ContainerStarted","Data":"c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020"} Oct 30 15:01:30.482899 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.482731 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" event={"ID":"b7f14ee3-e72a-49fe-97ee-769c14dd0824","Type":"ContainerStarted","Data":"2c81dcab9f3a0e73b1a8ea9b5a838f89fec3b7238576d4d0991035d2a23d2461"} Oct 30 15:01:30.484646 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.484618 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" event={"ID":"a801d113-37ee-4dbb-b972-9f93b36d6bd5","Type":"ContainerStarted","Data":"90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac"} Oct 30 15:01:30.484741 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.484683 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" event={"ID":"a801d113-37ee-4dbb-b972-9f93b36d6bd5","Type":"ContainerStarted","Data":"d82af5b1def6630942c15bc70284abe40d0b057545b70427202ad5b77727a8ef"} Oct 30 15:01:30.484741 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.484719 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:30.528947 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.528873 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" podStartSLOduration=2.156861313 podStartE2EDuration="2.528856864s" totalImagesPullingTime="371.995551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:30.527916387 +0000 UTC m=+4320.328421832" watchObservedRunningTime="2026-10-30 15:01:30.528856864 +0000 UTC m=+4320.329362309" Oct 30 15:01:30.574644 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.574586 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" podStartSLOduration=1.574574393 podStartE2EDuration="1.574574393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:30.572959457 +0000 UTC m=+4320.373464936" watchObservedRunningTime="2026-10-30 15:01:30.574574393 +0000 UTC m=+4320.375079870" Oct 30 15:01:30.619204 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:30.619136 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" podStartSLOduration=2.6191208379999997 podStartE2EDuration="2.619120838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:30.616568564 +0000 UTC m=+4320.417074008" watchObservedRunningTime="2026-10-30 15:01:30.619120838 +0000 UTC m=+4320.419626273" Oct 30 15:01:31.489572 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:31.489545 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:33.386951 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:33.386912 5824 scope.go:122] "RemoveContainer" containerID="df47071374a8bf9326037d255b1072643a549ca92cbadec1ab2aa756c66a96b0" Oct 30 15:01:33.395705 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:33.395680 5824 scope.go:122] "RemoveContainer" containerID="a1c67d9728d8b7d108319affe916cf1a11a5faaf5d835f6b2e99c092011e6379" Oct 30 15:01:33.667314 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:01:33.654453 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:01:36.494165 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:36.494132 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:01:41.493078 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:41.492999 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:01:42.498143 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:42.498115 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:01:42.549108 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:42.548913 5824 generic.go:356] "Generic (PLEG): container finished" podID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerID="d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5" exitCode=0 Oct 30 15:01:42.549108 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:42.548964 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" event={"ID":"c3d28589-20c5-42e1-85eb-80a26e4df08d","Type":"ContainerDied","Data":"d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5"} Oct 30 15:01:42.549471 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:42.549450 5824 scope.go:122] "RemoveContainer" containerID="d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5" Oct 30 15:01:42.550780 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:42.550744 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:01:42.550885 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:42.550838 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:01:42.550940 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:42.550888 5824 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:01:43.572182 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:43.572140 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" event={"ID":"c3d28589-20c5-42e1-85eb-80a26e4df08d","Type":"ContainerStarted","Data":"d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9"} Oct 30 15:01:43.871261 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:43.871133 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:43.872240 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:43.872215 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 15:01:43.872368 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:43.872276 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 15:01:44.876214 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:44.874271 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 15:01:44.876214 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:44.874330 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 15:01:45.871371 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:45.871331 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 15:01:45.871561 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:45.871405 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 15:01:46.403925 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:46.403888 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:46.871432 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:46.871394 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 15:01:46.871626 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:46.871458 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 15:01:47.871946 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:47.871903 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body= Oct 30 15:01:47.872507 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:47.871972 5824 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" Oct 30 15:01:49.176282 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:49.176247 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:01:49.177730 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:01:49.177699 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:02:03.641093 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:02:03.641051 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:02:33.422846 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:33.422809 5824 scope.go:122] "RemoveContainer" containerID="82dae7a7ae2ab21581359eabb6da399f59d6f9319a903ae9419e171b64b6339d" Oct 30 15:02:33.431542 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:33.431517 5824 scope.go:122] "RemoveContainer" containerID="add9503e6a07ef005b87484d5d3cf14a3bce2eb3109daeefd165cb52089e9f57" Oct 30 15:02:33.439256 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:33.439231 5824 scope.go:122] "RemoveContainer" containerID="3012dbcb1e99f3c8d55de594a674178d3ff4550f25df5429ecbb1b00cf2e3e5f" Oct 30 15:02:33.639591 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:02:33.639556 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:02:35.272027 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:35.271990 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm"] Oct 30 15:02:35.302877 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:35.302844 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j"] Oct 30 15:02:38.303100 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:38.303060 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d"] Oct 30 15:02:38.303802 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:38.303349 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" podUID="4572cf7a-17e3-4448-b265-143028f333ab" containerName="maas-api" containerID="cri-o://b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18" gracePeriod=30 Oct 30 15:02:39.552544 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.552519 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:02:39.670005 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.669916 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-metrics-certs\") pod \"4572cf7a-17e3-4448-b265-143028f333ab\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " Oct 30 15:02:39.670005 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.669957 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4572cf7a-17e3-4448-b265-143028f333ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4572cf7a-17e3-4448-b265-143028f333ab-openshift-service-ca\") pod \"4572cf7a-17e3-4448-b265-143028f333ab\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " Oct 30 15:02:39.670283 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.670043 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4572cf7a-17e3-4448-b265-143028f333ab-kube-api-access-q58l5\" (UniqueName: \"kubernetes.io/projected/4572cf7a-17e3-4448-b265-143028f333ab-kube-api-access-q58l5\") pod \"4572cf7a-17e3-4448-b265-143028f333ab\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " Oct 30 15:02:39.670283 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.670107 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-maas-api-tls\") pod \"4572cf7a-17e3-4448-b265-143028f333ab\" (UID: \"4572cf7a-17e3-4448-b265-143028f333ab\") " Oct 30 15:02:39.670445 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.670420 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4572cf7a-17e3-4448-b265-143028f333ab-openshift-service-ca" pod "4572cf7a-17e3-4448-b265-143028f333ab" (UID: "4572cf7a-17e3-4448-b265-143028f333ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:02:39.670527 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.670509 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4572cf7a-17e3-4448-b265-143028f333ab-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:39.672167 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.672141 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-metrics-certs" pod "4572cf7a-17e3-4448-b265-143028f333ab" (UID: "4572cf7a-17e3-4448-b265-143028f333ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:02:39.672289 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.672222 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4572cf7a-17e3-4448-b265-143028f333ab-kube-api-access-q58l5" pod "4572cf7a-17e3-4448-b265-143028f333ab" (UID: "4572cf7a-17e3-4448-b265-143028f333ab"). InnerVolumeSpecName "kube-api-access-q58l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:02:39.672335 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.672297 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-maas-api-tls" pod "4572cf7a-17e3-4448-b265-143028f333ab" (UID: "4572cf7a-17e3-4448-b265-143028f333ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:02:39.771061 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.771027 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:39.771061 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.771057 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q58l5\" (UniqueName: \"kubernetes.io/projected/4572cf7a-17e3-4448-b265-143028f333ab-kube-api-access-q58l5\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:39.771061 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.771068 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4572cf7a-17e3-4448-b265-143028f333ab-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:39.880058 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.880013 5824 generic.go:356] "Generic (PLEG): container finished" podID="4572cf7a-17e3-4448-b265-143028f333ab" containerID="b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18" exitCode=0 Oct 30 15:02:39.880260 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.880066 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" Oct 30 15:02:39.880260 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.880102 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" event={"ID":"4572cf7a-17e3-4448-b265-143028f333ab","Type":"ContainerDied","Data":"b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18"} Oct 30 15:02:39.880260 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.880172 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d" event={"ID":"4572cf7a-17e3-4448-b265-143028f333ab","Type":"ContainerDied","Data":"282bdf991dd8aadd9b4ef4bada3aaec7d47461c9bf3f2467f0ae59295c5f50aa"} Oct 30 15:02:39.880260 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.880215 5824 scope.go:122] "RemoveContainer" containerID="b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18" Oct 30 15:02:39.889734 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.889717 5824 scope.go:122] "RemoveContainer" containerID="b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18" Oct 30 15:02:39.889996 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:02:39.889975 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18\": container with ID starting with b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18 not found: ID does not exist" containerID="b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18" Oct 30 15:02:39.890043 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.890005 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18"} err="failed to get container status \"b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18\": rpc error: code = NotFound desc = could not find container \"b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18\": container with ID starting with b94f3d261a4fa86720cb4e2fe2234a1b8fe31297f675877d8086eafd50392f18 not found: ID does not exist" Oct 30 15:02:39.938956 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.938924 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d"] Oct 30 15:02:39.946987 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:39.946965 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ef0c7-548d86db68-c9l7d"] Oct 30 15:02:40.272571 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:40.272472 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" podUID="a801d113-37ee-4dbb-b972-9f93b36d6bd5" containerName="payload-processing" containerID="cri-o://90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac" gracePeriod=25 Oct 30 15:02:40.303179 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:40.303133 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" podUID="b7f14ee3-e72a-49fe-97ee-769c14dd0824" containerName="payload-pre-processing" containerID="cri-o://c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020" gracePeriod=25 Oct 30 15:02:40.315755 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:40.315726 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4572cf7a-17e3-4448-b265-143028f333ab" path="/var/lib/kubelet/pods/4572cf7a-17e3-4448-b265-143028f333ab/volumes" Oct 30 15:02:41.490580 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:41.490541 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" podUID="a801d113-37ee-4dbb-b972-9f93b36d6bd5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:02:42.496114 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:42.496077 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" podUID="b7f14ee3-e72a-49fe-97ee-769c14dd0824" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:02:46.852953 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:46.852914 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2"] Oct 30 15:02:46.853384 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:46.853148 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" containerID="cri-o://d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9" gracePeriod=30 Oct 30 15:02:49.178824 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:49.178785 5824 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:02:49.179237 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:49.178849 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:02:51.491055 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:51.491018 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" podUID="a801d113-37ee-4dbb-b972-9f93b36d6bd5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:02:52.104940 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.104915 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:02:52.202725 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.202691 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-certs\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.202725 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.202731 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-envoy\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.202957 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.202769 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d28589-20c5-42e1-85eb-80a26e4df08d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d28589-20c5-42e1-85eb-80a26e4df08d-istiod-ca-cert\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.202957 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.202829 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-token\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-token\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.202957 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.202878 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-credential-socket\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.202957 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.202922 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-data\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.203175 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.202998 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-podinfo\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.203175 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203014 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-certs" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:52.203175 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203037 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-socket\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.203175 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203082 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-kube-api-access-6v68r\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-kube-api-access-6v68r\") pod \"c3d28589-20c5-42e1-85eb-80a26e4df08d\" (UID: \"c3d28589-20c5-42e1-85eb-80a26e4df08d\") " Oct 30 15:02:52.203431 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203280 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-credential-socket" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:52.203431 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203327 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d28589-20c5-42e1-85eb-80a26e4df08d-istiod-ca-cert" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:02:52.203431 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203412 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-data" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:52.203592 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203568 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-socket" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:52.203659 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203636 5824 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.203659 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203653 5824 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d28589-20c5-42e1-85eb-80a26e4df08d-istiod-ca-cert\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.203779 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203662 5824 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-credential-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.203779 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.203671 5824 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-data\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.205333 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.205301 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-podinfo" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:02:52.205442 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.205355 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-kube-api-access-6v68r" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "kube-api-access-6v68r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:02:52.205532 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.205512 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-token" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:02:52.205657 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.205638 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-envoy" pod "c3d28589-20c5-42e1-85eb-80a26e4df08d" (UID: "c3d28589-20c5-42e1-85eb-80a26e4df08d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:02:52.304823 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.304784 5824 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-token\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.304823 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.304816 5824 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-podinfo\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.304823 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.304825 5824 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-workload-socket\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.305065 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.304835 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v68r\" (UniqueName: \"kubernetes.io/projected/c3d28589-20c5-42e1-85eb-80a26e4df08d-kube-api-access-6v68r\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.305065 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.304844 5824 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d28589-20c5-42e1-85eb-80a26e4df08d-istio-envoy\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:02:52.496557 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.496517 5824 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" podUID="b7f14ee3-e72a-49fe-97ee-769c14dd0824" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:02:52.950529 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.950496 5824 generic.go:356] "Generic (PLEG): container finished" podID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerID="d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9" exitCode=0 Oct 30 15:02:52.950736 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.950551 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" Oct 30 15:02:52.950736 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.950579 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" event={"ID":"c3d28589-20c5-42e1-85eb-80a26e4df08d","Type":"ContainerDied","Data":"d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9"} Oct 30 15:02:52.950736 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.950634 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2" event={"ID":"c3d28589-20c5-42e1-85eb-80a26e4df08d","Type":"ContainerDied","Data":"539d6675ac7735a2589d944dc97a60ebbc2d3bb6b2b05bf8d299ee5b1e0d195b"} Oct 30 15:02:52.950736 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.950658 5824 scope.go:122] "RemoveContainer" containerID="d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9" Oct 30 15:02:52.959542 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.959525 5824 scope.go:122] "RemoveContainer" containerID="d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5" Oct 30 15:02:52.967223 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.967181 5824 scope.go:122] "RemoveContainer" containerID="d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9" Oct 30 15:02:52.967476 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:02:52.967451 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9\": container with ID starting with d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9 not found: ID does not exist" containerID="d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9" Oct 30 15:02:52.967539 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.967477 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9"} err="failed to get container status \"d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9\": rpc error: code = NotFound desc = could not find container \"d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9\": container with ID starting with d7890b2dfb677e3f86ef6c3a6bfb3b0288f21a731c27b42d313861f5b4c0dbc9 not found: ID does not exist" Oct 30 15:02:52.967539 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.967495 5824 scope.go:122] "RemoveContainer" containerID="d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5" Oct 30 15:02:52.967752 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:02:52.967738 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5\": container with ID starting with d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5 not found: ID does not exist" containerID="d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5" Oct 30 15:02:52.967795 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:52.967755 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5"} err="failed to get container status \"d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5\": rpc error: code = NotFound desc = could not find container \"d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5\": container with ID starting with d5a509276356a4bf3225df19c3f1a5c79139b84d878b5b9b1ddde61f83ba66b5 not found: ID does not exist" Oct 30 15:02:53.022630 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:53.022607 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2"] Oct 30 15:02:53.036201 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:53.036173 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ef0c7-openshift-default-6f74b859b5lxdg2"] Oct 30 15:02:54.316268 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:02:54.316230 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" path="/var/lib/kubelet/pods/c3d28589-20c5-42e1-85eb-80a26e4df08d/volumes" Oct 30 15:03:00.563777 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.563747 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:03:00.590592 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.590571 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:03:00.715427 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.715396 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f14ee3-e72a-49fe-97ee-769c14dd0824-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f14ee3-e72a-49fe-97ee-769c14dd0824-plugins-config-volume\") pod \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\" (UID: \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\") " Oct 30 15:03:00.715604 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.715472 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a801d113-37ee-4dbb-b972-9f93b36d6bd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a801d113-37ee-4dbb-b972-9f93b36d6bd5-plugins-config-volume\") pod \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\" (UID: \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\") " Oct 30 15:03:00.715604 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.715556 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f14ee3-e72a-49fe-97ee-769c14dd0824-kube-api-access-wqh9b\" (UniqueName: \"kubernetes.io/projected/b7f14ee3-e72a-49fe-97ee-769c14dd0824-kube-api-access-wqh9b\") pod \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\" (UID: \"b7f14ee3-e72a-49fe-97ee-769c14dd0824\") " Oct 30 15:03:00.715715 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.715632 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a801d113-37ee-4dbb-b972-9f93b36d6bd5-kube-api-access-zkm56\" (UniqueName: \"kubernetes.io/projected/a801d113-37ee-4dbb-b972-9f93b36d6bd5-kube-api-access-zkm56\") pod \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\" (UID: \"a801d113-37ee-4dbb-b972-9f93b36d6bd5\") " Oct 30 15:03:00.715868 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.715847 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a801d113-37ee-4dbb-b972-9f93b36d6bd5-plugins-config-volume" pod "a801d113-37ee-4dbb-b972-9f93b36d6bd5" (UID: "a801d113-37ee-4dbb-b972-9f93b36d6bd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:00.715917 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.715846 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f14ee3-e72a-49fe-97ee-769c14dd0824-plugins-config-volume" pod "b7f14ee3-e72a-49fe-97ee-769c14dd0824" (UID: "b7f14ee3-e72a-49fe-97ee-769c14dd0824"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:00.716009 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.715993 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f14ee3-e72a-49fe-97ee-769c14dd0824-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:03:00.716069 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.716010 5824 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a801d113-37ee-4dbb-b972-9f93b36d6bd5-plugins-config-volume\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:03:00.717728 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.717702 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f14ee3-e72a-49fe-97ee-769c14dd0824-kube-api-access-wqh9b" pod "b7f14ee3-e72a-49fe-97ee-769c14dd0824" (UID: "b7f14ee3-e72a-49fe-97ee-769c14dd0824"). InnerVolumeSpecName "kube-api-access-wqh9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:00.717827 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.717792 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a801d113-37ee-4dbb-b972-9f93b36d6bd5-kube-api-access-zkm56" pod "a801d113-37ee-4dbb-b972-9f93b36d6bd5" (UID: "a801d113-37ee-4dbb-b972-9f93b36d6bd5"). InnerVolumeSpecName "kube-api-access-zkm56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:00.817255 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.817134 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqh9b\" (UniqueName: \"kubernetes.io/projected/b7f14ee3-e72a-49fe-97ee-769c14dd0824-kube-api-access-wqh9b\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:03:00.817255 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:00.817178 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkm56\" (UniqueName: \"kubernetes.io/projected/a801d113-37ee-4dbb-b972-9f93b36d6bd5-kube-api-access-zkm56\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:03:01.025273 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.025239 5824 generic.go:356] "Generic (PLEG): container finished" podID="a801d113-37ee-4dbb-b972-9f93b36d6bd5" containerID="90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac" exitCode=0 Oct 30 15:03:01.025436 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.025292 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" Oct 30 15:03:01.025436 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.025330 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" event={"ID":"a801d113-37ee-4dbb-b972-9f93b36d6bd5","Type":"ContainerDied","Data":"90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac"} Oct 30 15:03:01.025436 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.025383 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm" event={"ID":"a801d113-37ee-4dbb-b972-9f93b36d6bd5","Type":"ContainerDied","Data":"d82af5b1def6630942c15bc70284abe40d0b057545b70427202ad5b77727a8ef"} Oct 30 15:03:01.025436 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.025428 5824 scope.go:122] "RemoveContainer" containerID="90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac" Oct 30 15:03:01.026821 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.026798 5824 generic.go:356] "Generic (PLEG): container finished" podID="b7f14ee3-e72a-49fe-97ee-769c14dd0824" containerID="c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020" exitCode=0 Oct 30 15:03:01.026893 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.026840 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" Oct 30 15:03:01.026935 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.026888 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" event={"ID":"b7f14ee3-e72a-49fe-97ee-769c14dd0824","Type":"ContainerDied","Data":"c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020"} Oct 30 15:03:01.026986 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.026945 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j" event={"ID":"b7f14ee3-e72a-49fe-97ee-769c14dd0824","Type":"ContainerDied","Data":"2c81dcab9f3a0e73b1a8ea9b5a838f89fec3b7238576d4d0991035d2a23d2461"} Oct 30 15:03:01.035122 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.035095 5824 scope.go:122] "RemoveContainer" containerID="90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac" Oct 30 15:03:01.035416 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:03:01.035397 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac\": container with ID starting with 90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac not found: ID does not exist" containerID="90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac" Oct 30 15:03:01.035467 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.035426 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac"} err="failed to get container status \"90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac\": rpc error: code = NotFound desc = could not find container \"90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac\": container with ID starting with 90a3c7a5c15b704a3ef1501d9ab68168bf38267b6147a0e1962c547fc82ba4ac not found: ID does not exist" Oct 30 15:03:01.035467 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.035444 5824 scope.go:122] "RemoveContainer" containerID="c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020" Oct 30 15:03:01.043733 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.043713 5824 scope.go:122] "RemoveContainer" containerID="c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020" Oct 30 15:03:01.043972 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:03:01.043956 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020\": container with ID starting with c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020 not found: ID does not exist" containerID="c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020" Oct 30 15:03:01.044023 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.043977 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020"} err="failed to get container status \"c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020\": rpc error: code = NotFound desc = could not find container \"c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020\": container with ID starting with c3b0a5be7f5fdbc5deec1fe6dc452574b1d82b39c459ebcee37b194a9b5ec020 not found: ID does not exist" Oct 30 15:03:01.083617 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.083562 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j"] Oct 30 15:03:01.088889 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.088870 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ef0c7-7466df4x8j"] Oct 30 15:03:01.127619 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.127593 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm"] Oct 30 15:03:01.130415 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:01.130389 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ef0c7-6f45f67cdvqcsm"] Oct 30 15:03:02.314872 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:02.314836 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a801d113-37ee-4dbb-b972-9f93b36d6bd5" path="/var/lib/kubelet/pods/a801d113-37ee-4dbb-b972-9f93b36d6bd5/volumes" Oct 30 15:03:02.315325 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:02.315309 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f14ee3-e72a-49fe-97ee-769c14dd0824" path="/var/lib/kubelet/pods/b7f14ee3-e72a-49fe-97ee-769c14dd0824/volumes" Oct 30 15:03:03.640850 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:03:03.640804 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:03:18.035461 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.035382 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws"] Oct 30 15:03:18.048462 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.048434 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d41b8de-n5jws"] Oct 30 15:03:18.243086 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243050 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw"] Oct 30 15:03:18.243543 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243528 5824 cpu_manager.go:414] "containerMap: removing container" podUID="4572cf7a-17e3-4448-b265-143028f333ab" containerName="maas-api" Oct 30 15:03:18.243588 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243545 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4572cf7a-17e3-4448-b265-143028f333ab" containerName="maas-api" Oct 30 15:03:18.243588 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243557 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" Oct 30 15:03:18.243588 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243562 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" Oct 30 15:03:18.243588 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243573 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b7f14ee3-e72a-49fe-97ee-769c14dd0824" containerName="payload-pre-processing" Oct 30 15:03:18.243588 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243578 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f14ee3-e72a-49fe-97ee-769c14dd0824" containerName="payload-pre-processing" Oct 30 15:03:18.243762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243595 5824 cpu_manager.go:414] "containerMap: removing container" podUID="a801d113-37ee-4dbb-b972-9f93b36d6bd5" containerName="payload-processing" Oct 30 15:03:18.243762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243600 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a801d113-37ee-4dbb-b972-9f93b36d6bd5" containerName="payload-processing" Oct 30 15:03:18.243762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243664 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="4572cf7a-17e3-4448-b265-143028f333ab" containerName="maas-api" Oct 30 15:03:18.243762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243673 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" Oct 30 15:03:18.243762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243679 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f14ee3-e72a-49fe-97ee-769c14dd0824" containerName="payload-pre-processing" Oct 30 15:03:18.243762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243685 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" Oct 30 15:03:18.243762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.243692 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="a801d113-37ee-4dbb-b972-9f93b36d6bd5" containerName="payload-processing" Oct 30 15:03:18.246784 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.246767 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.265017 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.264990 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7f7d1aa4-kserve-self-signed-certs\"" Oct 30 15:03:18.265663 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.265641 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw"] Oct 30 15:03:18.275689 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.275663 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tls-certs\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.275799 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.275704 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-dshm\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.275799 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.275783 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tmp-dir\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.276012 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.275984 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fgk\" (UniqueName: \"kubernetes.io/projected/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-kube-api-access-l9fgk\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.276059 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.276043 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-model-cache\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.276103 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.276080 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-home\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.315656 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.315590 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07687d7b-87ee-4bef-949f-1e89f05b6a3f" path="/var/lib/kubelet/pods/07687d7b-87ee-4bef-949f-1e89f05b6a3f/volumes" Oct 30 15:03:18.376970 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.376944 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fgk\" (UniqueName: \"kubernetes.io/projected/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-kube-api-access-l9fgk\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.377123 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.376982 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-model-cache\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.377213 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.377122 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-home\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.377341 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.377216 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tls-certs\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.377341 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.377302 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-dshm\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.377446 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.377346 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tmp-dir\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.377671 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.377644 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-model-cache\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.377780 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.377732 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-home\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.377979 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.377956 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tmp-dir\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.379485 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.379460 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-dshm\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.380060 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.380043 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tls-certs\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.402755 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.402728 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fgk\" (UniqueName: \"kubernetes.io/projected/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-kube-api-access-l9fgk\") pod \"e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.559743 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.559705 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:18.862478 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:18.862448 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw"] Oct 30 15:03:18.862986 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:03:18.862965 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb46033c2_2e18_4d9c_ad5c_1c6d39cdcb84.slice/crio-a8f0071db7f7b246a69a3d220b7854629aeab59dd461f0db3f762ae08fe475e4 WatchSource:0}: Error finding container a8f0071db7f7b246a69a3d220b7854629aeab59dd461f0db3f762ae08fe475e4: Status 404 returned error can't find the container with id a8f0071db7f7b246a69a3d220b7854629aeab59dd461f0db3f762ae08fe475e4 Oct 30 15:03:19.119147 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:19.119037 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" event={"ID":"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84","Type":"ContainerStarted","Data":"2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee"} Oct 30 15:03:19.119147 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:19.119084 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" event={"ID":"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84","Type":"ContainerStarted","Data":"a8f0071db7f7b246a69a3d220b7854629aeab59dd461f0db3f762ae08fe475e4"} Oct 30 15:03:19.164237 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:19.164164 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" podStartSLOduration=1.164147267 podStartE2EDuration="1.164147267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:19.163464807 +0000 UTC m=+4428.963970252" watchObservedRunningTime="2026-10-30 15:03:19.164147267 +0000 UTC m=+4428.964652710" Oct 30 15:03:19.560608 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:19.560573 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:28.560145 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:28.560102 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:28.572657 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:28.572634 5824 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:29.195344 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:29.195308 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:03:33.465321 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:33.465289 5824 scope.go:122] "RemoveContainer" containerID="94effb9ba8ebb6b90c60a8ae38704b750eec2ddbb55515c6396be9a27199e106" Oct 30 15:03:33.639754 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:03:33.639714 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:03:44.944783 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:44.944749 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252"] Oct 30 15:03:44.945229 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:44.945158 5824 cpu_manager.go:414] "containerMap: removing container" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" Oct 30 15:03:44.945229 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:44.945169 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d28589-20c5-42e1-85eb-80a26e4df08d" containerName="istio-proxy" Oct 30 15:03:44.948294 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:44.948277 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:44.968216 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:44.968172 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252"] Oct 30 15:03:44.979724 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:44.979697 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:44.979831 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:44.979809 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:03:45.012701 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.012681 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:03:45.012865 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.012847 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:03:45.065481 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.065457 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.065603 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.065524 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.065647 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.065620 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2jk\" (UniqueName: \"kubernetes.io/projected/2d2e6988-4032-4495-8a42-c293bb0abbd0-kube-api-access-2x2jk\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.166241 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.166215 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.166368 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.166289 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.166368 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.166312 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2jk\" (UniqueName: \"kubernetes.io/projected/2d2e6988-4032-4495-8a42-c293bb0abbd0-kube-api-access-2x2jk\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.166940 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.166914 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.168445 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.168422 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.191968 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.191948 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2jk\" (UniqueName: \"kubernetes.io/projected/2d2e6988-4032-4495-8a42-c293bb0abbd0-kube-api-access-2x2jk\") pod \"maas-api-revoke-sub-452b6d7b-9p252\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.260495 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.260435 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:45.557410 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:45.557378 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252"] Oct 30 15:03:45.557561 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:03:45.557510 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2e6988_4032_4495_8a42_c293bb0abbd0.slice/crio-997e946e676af357a9fcf04c71f5114b563a2c3a808300044fea8686d42ab962 WatchSource:0}: Error finding container 997e946e676af357a9fcf04c71f5114b563a2c3a808300044fea8686d42ab962: Status 404 returned error can't find the container with id 997e946e676af357a9fcf04c71f5114b563a2c3a808300044fea8686d42ab962 Oct 30 15:03:46.265928 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:46.265889 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" event={"ID":"2d2e6988-4032-4495-8a42-c293bb0abbd0","Type":"ContainerStarted","Data":"473071bde90bfc4a140722dde2f0c7f81611091a9526783eb4f635c4dbe3c1c1"} Oct 30 15:03:46.266320 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:46.265944 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" event={"ID":"2d2e6988-4032-4495-8a42-c293bb0abbd0","Type":"ContainerStarted","Data":"997e946e676af357a9fcf04c71f5114b563a2c3a808300044fea8686d42ab962"} Oct 30 15:03:46.311825 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:46.311771 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" podStartSLOduration=2.311754395 podStartE2EDuration="2.311754395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:46.309626922 +0000 UTC m=+4456.110132368" watchObservedRunningTime="2026-10-30 15:03:46.311754395 +0000 UTC m=+4456.112259840" Oct 30 15:03:47.274712 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:47.274669 5824 generic.go:356] "Generic (PLEG): container finished" podID="2d2e6988-4032-4495-8a42-c293bb0abbd0" containerID="473071bde90bfc4a140722dde2f0c7f81611091a9526783eb4f635c4dbe3c1c1" exitCode=0 Oct 30 15:03:47.274712 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:47.274715 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" event={"ID":"2d2e6988-4032-4495-8a42-c293bb0abbd0","Type":"ContainerDied","Data":"473071bde90bfc4a140722dde2f0c7f81611091a9526783eb4f635c4dbe3c1c1"} Oct 30 15:03:48.712272 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.711144 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:48.731442 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.731404 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2e6988-4032-4495-8a42-c293bb0abbd0-kube-api-access-2x2jk\" (UniqueName: \"kubernetes.io/projected/2d2e6988-4032-4495-8a42-c293bb0abbd0-kube-api-access-2x2jk\") pod \"2d2e6988-4032-4495-8a42-c293bb0abbd0\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " Oct 30 15:03:48.731632 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.731506 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-openshift-ca\") pod \"2d2e6988-4032-4495-8a42-c293bb0abbd0\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " Oct 30 15:03:48.731632 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.731563 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-xks-ca\") pod \"2d2e6988-4032-4495-8a42-c293bb0abbd0\" (UID: \"2d2e6988-4032-4495-8a42-c293bb0abbd0\") " Oct 30 15:03:48.732967 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.732904 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-openshift-ca" pod "2d2e6988-4032-4495-8a42-c293bb0abbd0" (UID: "2d2e6988-4032-4495-8a42-c293bb0abbd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:48.735108 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.735071 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-xks-ca" pod "2d2e6988-4032-4495-8a42-c293bb0abbd0" (UID: "2d2e6988-4032-4495-8a42-c293bb0abbd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:48.735236 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.735074 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2e6988-4032-4495-8a42-c293bb0abbd0-kube-api-access-2x2jk" pod "2d2e6988-4032-4495-8a42-c293bb0abbd0" (UID: "2d2e6988-4032-4495-8a42-c293bb0abbd0"). InnerVolumeSpecName "kube-api-access-2x2jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:48.833009 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.832969 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2jk\" (UniqueName: \"kubernetes.io/projected/2d2e6988-4032-4495-8a42-c293bb0abbd0-kube-api-access-2x2jk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:03:48.833009 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.833003 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:03:48.833009 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:48.833014 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2e6988-4032-4495-8a42-c293bb0abbd0-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:03:49.310232 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:49.308258 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" event={"ID":"2d2e6988-4032-4495-8a42-c293bb0abbd0","Type":"ContainerDied","Data":"997e946e676af357a9fcf04c71f5114b563a2c3a808300044fea8686d42ab962"} Oct 30 15:03:49.310232 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:49.308425 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="997e946e676af357a9fcf04c71f5114b563a2c3a808300044fea8686d42ab962" Oct 30 15:03:49.310232 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:49.308359 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252" Oct 30 15:03:50.034687 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:50.034651 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2"] Oct 30 15:03:50.053577 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:50.053526 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j95p2"] Oct 30 15:03:50.314866 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:03:50.314788 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9687edf9-7a77-4f59-83e6-b9e87e4ce4df" path="/var/lib/kubelet/pods/9687edf9-7a77-4f59-83e6-b9e87e4ce4df/volumes" Oct 30 15:04:00.606635 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:00.606590 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw"] Oct 30 15:04:03.645215 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:04:03.643769 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:04:05.557177 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.557144 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q"] Oct 30 15:04:05.557610 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.557595 5824 cpu_manager.go:414] "containerMap: removing container" podUID="2d2e6988-4032-4495-8a42-c293bb0abbd0" containerName="revoke-keys" Oct 30 15:04:05.557656 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.557611 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2e6988-4032-4495-8a42-c293bb0abbd0" containerName="revoke-keys" Oct 30 15:04:05.557693 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.557676 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2e6988-4032-4495-8a42-c293bb0abbd0" containerName="revoke-keys" Oct 30 15:04:05.561887 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.561870 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.579479 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.579452 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q"] Oct 30 15:04:05.590752 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.590733 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:05.590919 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.590907 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:05.590999 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.590979 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:05.592254 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.592235 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:04:05.607484 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.607455 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.607593 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.607501 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.607649 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.607588 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw274\" (UniqueName: \"kubernetes.io/projected/71a47e6d-43c2-4861-affa-73e5dae45cf0-kube-api-access-tw274\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.708158 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.708130 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.708329 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.708172 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.708329 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.708219 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw274\" (UniqueName: \"kubernetes.io/projected/71a47e6d-43c2-4861-affa-73e5dae45cf0-kube-api-access-tw274\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.708911 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.708891 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.710524 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.710500 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.734268 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.734244 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw274\" (UniqueName: \"kubernetes.io/projected/71a47e6d-43c2-4861-affa-73e5dae45cf0-kube-api-access-tw274\") pod \"maas-api-revoke-sub-3e31a506-4kt7q\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:05.875000 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:05.874920 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:06.172530 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:06.172500 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q"] Oct 30 15:04:06.173311 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:04:06.173281 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a47e6d_43c2_4861_affa_73e5dae45cf0.slice/crio-a3c1f9d305694d40610061371c2d0ad22f34b7f2f27bd45dcfa4c06eb38ca08d WatchSource:0}: Error finding container a3c1f9d305694d40610061371c2d0ad22f34b7f2f27bd45dcfa4c06eb38ca08d: Status 404 returned error can't find the container with id a3c1f9d305694d40610061371c2d0ad22f34b7f2f27bd45dcfa4c06eb38ca08d Oct 30 15:04:06.445122 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:06.445090 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" event={"ID":"71a47e6d-43c2-4861-affa-73e5dae45cf0","Type":"ContainerStarted","Data":"a3c1f9d305694d40610061371c2d0ad22f34b7f2f27bd45dcfa4c06eb38ca08d"} Oct 30 15:04:07.453425 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:07.453386 5824 generic.go:356] "Generic (PLEG): container finished" podID="71a47e6d-43c2-4861-affa-73e5dae45cf0" containerID="4c2ebac8704c9941ff5f7c01e4bc1f18f6f7e5aed6548963bb110ddd2fda75c7" exitCode=0 Oct 30 15:04:07.453963 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:07.453491 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" event={"ID":"71a47e6d-43c2-4861-affa-73e5dae45cf0","Type":"ContainerDied","Data":"4c2ebac8704c9941ff5f7c01e4bc1f18f6f7e5aed6548963bb110ddd2fda75c7"} Oct 30 15:04:08.853716 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:08.853382 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:08.937349 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:08.937315 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-xks-ca\") pod \"71a47e6d-43c2-4861-affa-73e5dae45cf0\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " Oct 30 15:04:08.937546 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:08.937386 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-openshift-ca\") pod \"71a47e6d-43c2-4861-affa-73e5dae45cf0\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " Oct 30 15:04:08.937632 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:08.937592 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a47e6d-43c2-4861-affa-73e5dae45cf0-kube-api-access-tw274\" (UniqueName: \"kubernetes.io/projected/71a47e6d-43c2-4861-affa-73e5dae45cf0-kube-api-access-tw274\") pod \"71a47e6d-43c2-4861-affa-73e5dae45cf0\" (UID: \"71a47e6d-43c2-4861-affa-73e5dae45cf0\") " Oct 30 15:04:08.937835 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:08.937803 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-openshift-ca" pod "71a47e6d-43c2-4861-affa-73e5dae45cf0" (UID: "71a47e6d-43c2-4861-affa-73e5dae45cf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:08.937999 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:08.937978 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:08.939707 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:08.939683 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-xks-ca" pod "71a47e6d-43c2-4861-affa-73e5dae45cf0" (UID: "71a47e6d-43c2-4861-affa-73e5dae45cf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:08.939956 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:08.939928 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a47e6d-43c2-4861-affa-73e5dae45cf0-kube-api-access-tw274" pod "71a47e6d-43c2-4861-affa-73e5dae45cf0" (UID: "71a47e6d-43c2-4861-affa-73e5dae45cf0"). InnerVolumeSpecName "kube-api-access-tw274". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:09.039326 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:09.039291 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw274\" (UniqueName: \"kubernetes.io/projected/71a47e6d-43c2-4861-affa-73e5dae45cf0-kube-api-access-tw274\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:09.039326 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:09.039322 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a47e6d-43c2-4861-affa-73e5dae45cf0-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:09.481383 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:09.481354 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" Oct 30 15:04:09.481553 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:09.481384 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q" event={"ID":"71a47e6d-43c2-4861-affa-73e5dae45cf0","Type":"ContainerDied","Data":"a3c1f9d305694d40610061371c2d0ad22f34b7f2f27bd45dcfa4c06eb38ca08d"} Oct 30 15:04:09.481553 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:09.481433 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3c1f9d305694d40610061371c2d0ad22f34b7f2f27bd45dcfa4c06eb38ca08d" Oct 30 15:04:15.721553 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:15.721486 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" podUID="b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" containerName="main" containerID="cri-o://2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee" gracePeriod=45 Oct 30 15:04:15.969081 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:15.969052 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:04:16.001684 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.001614 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-model-cache\") pod \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " Oct 30 15:04:16.001684 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.001652 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-dshm\") pod \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " Oct 30 15:04:16.001684 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.001683 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-home\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-home\") pod \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " Oct 30 15:04:16.001919 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.001765 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-kube-api-access-l9fgk\" (UniqueName: \"kubernetes.io/projected/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-kube-api-access-l9fgk\") pod \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " Oct 30 15:04:16.001919 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.001893 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tmp-dir\") pod \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " Oct 30 15:04:16.002009 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.001917 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-model-cache" pod "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" (UID: "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:04:16.002009 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.001933 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tls-certs\" (UniqueName: \"kubernetes.io/secret/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tls-certs\") pod \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\" (UID: \"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84\") " Oct 30 15:04:16.002009 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.001946 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-home" pod "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" (UID: "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:04:16.002165 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.002118 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tmp-dir" pod "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" (UID: "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:04:16.002304 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.002285 5824 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tmp-dir\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:16.002304 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.002305 5824 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-model-cache\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:16.002434 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.002314 5824 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-home\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:16.003966 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.003942 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-dshm" pod "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" (UID: "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:04:16.004082 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.003987 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tls-certs" pod "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" (UID: "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:16.004082 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.004029 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-kube-api-access-l9fgk" pod "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" (UID: "b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84"). InnerVolumeSpecName "kube-api-access-l9fgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:16.103079 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.103039 5824 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-tls-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:16.103079 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.103072 5824 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-dshm\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:16.103311 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.103086 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fgk\" (UniqueName: \"kubernetes.io/projected/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84-kube-api-access-l9fgk\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:16.515006 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.514972 5824 generic.go:356] "Generic (PLEG): container finished" podID="b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" containerID="2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee" exitCode=0 Oct 30 15:04:16.515226 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.515007 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" event={"ID":"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84","Type":"ContainerDied","Data":"2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee"} Oct 30 15:04:16.515226 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.515020 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" Oct 30 15:04:16.515226 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.515056 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw" event={"ID":"b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84","Type":"ContainerDied","Data":"a8f0071db7f7b246a69a3d220b7854629aeab59dd461f0db3f762ae08fe475e4"} Oct 30 15:04:16.515226 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.515073 5824 scope.go:122] "RemoveContainer" containerID="2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee" Oct 30 15:04:16.523864 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.523845 5824 scope.go:122] "RemoveContainer" containerID="2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee" Oct 30 15:04:16.524144 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:04:16.524121 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee\": container with ID starting with 2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee not found: ID does not exist" containerID="2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee" Oct 30 15:04:16.524219 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.524155 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee"} err="failed to get container status \"2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee\": rpc error: code = NotFound desc = could not find container \"2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee\": container with ID starting with 2257dac2b020f54d3455569fc3e07b6719482642d4a707a6769f8f9afde72bee not found: ID does not exist" Oct 30 15:04:16.572081 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.572051 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw"] Oct 30 15:04:16.585610 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:16.585542 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7f7d1aa4-kserve-6ff648c98f-wh8rw"] Oct 30 15:04:18.315698 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:18.315663 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" path="/var/lib/kubelet/pods/b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84/volumes" Oct 30 15:04:28.105981 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.105947 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s"] Oct 30 15:04:28.106450 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.106434 5824 cpu_manager.go:414] "containerMap: removing container" podUID="b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" containerName="main" Oct 30 15:04:28.106496 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.106450 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" containerName="main" Oct 30 15:04:28.106496 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.106471 5824 cpu_manager.go:414] "containerMap: removing container" podUID="71a47e6d-43c2-4861-affa-73e5dae45cf0" containerName="revoke-keys" Oct 30 15:04:28.106496 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.106476 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a47e6d-43c2-4861-affa-73e5dae45cf0" containerName="revoke-keys" Oct 30 15:04:28.106604 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.106546 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a47e6d-43c2-4861-affa-73e5dae45cf0" containerName="revoke-keys" Oct 30 15:04:28.106604 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.106564 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46033c2-2e18-4d9c-ad5c-1c6d39cdcb84" containerName="main" Oct 30 15:04:28.115300 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.115266 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.131155 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.131118 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s"] Oct 30 15:04:28.142517 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.142488 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:28.142517 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.142488 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:28.142723 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.142529 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:28.146316 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.146295 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgstp\"" Oct 30 15:04:28.157714 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.157680 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75z4v\" (UniqueName: \"kubernetes.io/projected/daf21a88-038d-44c1-9619-085949318f0c-kube-api-access-75z4v\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.157860 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.157734 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf21a88-038d-44c1-9619-085949318f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.157932 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.157906 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf21a88-038d-44c1-9619-085949318f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.259447 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.259413 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf21a88-038d-44c1-9619-085949318f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.259642 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.259516 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf21a88-038d-44c1-9619-085949318f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.259642 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.259551 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75z4v\" (UniqueName: \"kubernetes.io/projected/daf21a88-038d-44c1-9619-085949318f0c-kube-api-access-75z4v\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.260222 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.260168 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf21a88-038d-44c1-9619-085949318f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.261791 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.261768 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf21a88-038d-44c1-9619-085949318f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.286288 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.286254 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75z4v\" (UniqueName: \"kubernetes.io/projected/daf21a88-038d-44c1-9619-085949318f0c-kube-api-access-75z4v\") pod \"maas-api-revoke-sub-31fef4f7-jvg8s\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.427991 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.427957 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:28.726855 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:28.726823 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s"] Oct 30 15:04:28.728122 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:04:28.728097 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf21a88_038d_44c1_9619_085949318f0c.slice/crio-42cedd4e2847bbae9c5e3c259843fcf95408acae29ddd3e44e0514d54c32281a WatchSource:0}: Error finding container 42cedd4e2847bbae9c5e3c259843fcf95408acae29ddd3e44e0514d54c32281a: Status 404 returned error can't find the container with id 42cedd4e2847bbae9c5e3c259843fcf95408acae29ddd3e44e0514d54c32281a Oct 30 15:04:29.594488 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:29.594454 5824 generic.go:356] "Generic (PLEG): container finished" podID="daf21a88-038d-44c1-9619-085949318f0c" containerID="91ce3e299704ea4cca96c39cc8d947a2966e6bdaabc10c75c120cf7ead1ee62d" exitCode=0 Oct 30 15:04:29.594880 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:29.594531 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" event={"ID":"daf21a88-038d-44c1-9619-085949318f0c","Type":"ContainerDied","Data":"91ce3e299704ea4cca96c39cc8d947a2966e6bdaabc10c75c120cf7ead1ee62d"} Oct 30 15:04:29.594880 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:29.594583 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" event={"ID":"daf21a88-038d-44c1-9619-085949318f0c","Type":"ContainerStarted","Data":"42cedd4e2847bbae9c5e3c259843fcf95408acae29ddd3e44e0514d54c32281a"} Oct 30 15:04:30.736404 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.736379 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:30.792173 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.792138 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf21a88-038d-44c1-9619-085949318f0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf21a88-038d-44c1-9619-085949318f0c-maas-api-openshift-ca\") pod \"daf21a88-038d-44c1-9619-085949318f0c\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " Oct 30 15:04:30.792357 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.792255 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf21a88-038d-44c1-9619-085949318f0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf21a88-038d-44c1-9619-085949318f0c-maas-api-xks-ca\") pod \"daf21a88-038d-44c1-9619-085949318f0c\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " Oct 30 15:04:30.792357 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.792328 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf21a88-038d-44c1-9619-085949318f0c-kube-api-access-75z4v\" (UniqueName: \"kubernetes.io/projected/daf21a88-038d-44c1-9619-085949318f0c-kube-api-access-75z4v\") pod \"daf21a88-038d-44c1-9619-085949318f0c\" (UID: \"daf21a88-038d-44c1-9619-085949318f0c\") " Oct 30 15:04:30.792567 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.792537 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf21a88-038d-44c1-9619-085949318f0c-maas-api-openshift-ca" pod "daf21a88-038d-44c1-9619-085949318f0c" (UID: "daf21a88-038d-44c1-9619-085949318f0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:30.792711 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.792689 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf21a88-038d-44c1-9619-085949318f0c-maas-api-openshift-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:30.794452 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.794423 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf21a88-038d-44c1-9619-085949318f0c-maas-api-xks-ca" pod "daf21a88-038d-44c1-9619-085949318f0c" (UID: "daf21a88-038d-44c1-9619-085949318f0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:30.794541 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.794461 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf21a88-038d-44c1-9619-085949318f0c-kube-api-access-75z4v" pod "daf21a88-038d-44c1-9619-085949318f0c" (UID: "daf21a88-038d-44c1-9619-085949318f0c"). InnerVolumeSpecName "kube-api-access-75z4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:30.894003 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.893914 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf21a88-038d-44c1-9619-085949318f0c-maas-api-xks-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:30.894003 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:30.893946 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75z4v\" (UniqueName: \"kubernetes.io/projected/daf21a88-038d-44c1-9619-085949318f0c-kube-api-access-75z4v\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:31.605576 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:31.605546 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" Oct 30 15:04:31.605784 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:31.605546 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s" event={"ID":"daf21a88-038d-44c1-9619-085949318f0c","Type":"ContainerDied","Data":"42cedd4e2847bbae9c5e3c259843fcf95408acae29ddd3e44e0514d54c32281a"} Oct 30 15:04:31.605784 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:31.605671 5824 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42cedd4e2847bbae9c5e3c259843fcf95408acae29ddd3e44e0514d54c32281a" Oct 30 15:04:33.502046 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:33.502013 5824 scope.go:122] "RemoveContainer" containerID="adc17b6f03c262260c8648b9851bf550a66585cb3549bca87f32856bcac765d6" Oct 30 15:04:33.510666 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:33.510647 5824 scope.go:122] "RemoveContainer" containerID="d8e83b8e5cd8d90b457c15ad3a2d259d31413a05998c90450c208db781035c74" Oct 30 15:04:33.640033 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:04:33.639999 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:04:40.412373 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.412342 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q"] Oct 30 15:04:40.412790 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.412774 5824 cpu_manager.go:414] "containerMap: removing container" podUID="daf21a88-038d-44c1-9619-085949318f0c" containerName="revoke-keys" Oct 30 15:04:40.412837 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.412792 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf21a88-038d-44c1-9619-085949318f0c" containerName="revoke-keys" Oct 30 15:04:40.412883 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.412869 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf21a88-038d-44c1-9619-085949318f0c" containerName="revoke-keys" Oct 30 15:04:40.415953 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.415928 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.434491 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.434464 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2pczd"] Oct 30 15:04:40.438434 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.438413 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.458521 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.448453 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q"] Oct 30 15:04:40.458521 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.448656 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 15:04:40.458521 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.448672 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 15:04:40.458521 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.448700 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-54whp\"" Oct 30 15:04:40.465590 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.465546 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2pczd"] Oct 30 15:04:40.522292 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.522263 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:40.522292 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.522288 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:40.585803 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.585778 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-openshift-service-ca\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.585919 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.585818 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-metrics-certs\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.585919 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.585839 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfrld\" (UniqueName: \"kubernetes.io/projected/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-kube-api-access-cfrld\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.586005 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.585957 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-maas-api-tls\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.586005 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.585992 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-maas-api-tls\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.586077 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.586056 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-metrics-certs\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.586152 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.586137 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q2sf\" (UniqueName: \"kubernetes.io/projected/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-kube-api-access-4q2sf\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.586218 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.586168 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-openshift-service-ca\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.687033 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687010 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q2sf\" (UniqueName: \"kubernetes.io/projected/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-kube-api-access-4q2sf\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.687145 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687043 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-openshift-service-ca\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.687145 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687071 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-openshift-service-ca\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.687145 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687106 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-metrics-certs\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.687145 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687121 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfrld\" (UniqueName: \"kubernetes.io/projected/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-kube-api-access-cfrld\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.687384 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687144 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-maas-api-tls\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.687384 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687174 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-maas-api-tls\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.687384 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687246 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-metrics-certs\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.687893 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687860 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-openshift-service-ca\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.688021 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.687945 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-openshift-service-ca\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.689793 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.689763 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-metrics-certs\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.689793 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.689781 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-maas-api-tls\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.690008 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.689990 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-metrics-certs\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.690074 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.690029 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-maas-api-tls\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.712700 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.712679 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfrld\" (UniqueName: \"kubernetes.io/projected/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-kube-api-access-cfrld\") pod \"maas-api-5c48877c75-jdr4q\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.712820 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.712802 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q2sf\" (UniqueName: \"kubernetes.io/projected/df9b2fc0-487d-47e6-b5e4-de49add2c8ae-kube-api-access-4q2sf\") pod \"maas-api-595cc648c-2pczd\" (UID: \"df9b2fc0-487d-47e6-b5e4-de49add2c8ae\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:40.733636 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.733613 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:40.755818 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:40.755799 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:41.044608 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:04:41.044569 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b48858d_e7c4_4cc9_aa2a_f1c07e6fc662.slice/crio-6a5c728e0ad66a7fd950aa775d47c0d53b53f2bd2c7994c346755ca8a09bde37 WatchSource:0}: Error finding container 6a5c728e0ad66a7fd950aa775d47c0d53b53f2bd2c7994c346755ca8a09bde37: Status 404 returned error can't find the container with id 6a5c728e0ad66a7fd950aa775d47c0d53b53f2bd2c7994c346755ca8a09bde37 Oct 30 15:04:41.045286 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.045259 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q"] Oct 30 15:04:41.066955 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:04:41.066926 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9b2fc0_487d_47e6_b5e4_de49add2c8ae.slice/crio-8fec96ecb5343595a514fc02c251d763397b287ada659127c779a1a83d4aa3b1 WatchSource:0}: Error finding container 8fec96ecb5343595a514fc02c251d763397b287ada659127c779a1a83d4aa3b1: Status 404 returned error can't find the container with id 8fec96ecb5343595a514fc02c251d763397b287ada659127c779a1a83d4aa3b1 Oct 30 15:04:41.068284 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.068251 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-2pczd"] Oct 30 15:04:41.658972 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.658941 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" event={"ID":"df9b2fc0-487d-47e6-b5e4-de49add2c8ae","Type":"ContainerStarted","Data":"036cf6f3a72929d556b8cbd40cda6816b189252a7cb3b4691ec4888e6f92f594"} Oct 30 15:04:41.659470 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.659008 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:41.659470 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.659031 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" event={"ID":"df9b2fc0-487d-47e6-b5e4-de49add2c8ae","Type":"ContainerStarted","Data":"8fec96ecb5343595a514fc02c251d763397b287ada659127c779a1a83d4aa3b1"} Oct 30 15:04:41.660618 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.660588 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" event={"ID":"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662","Type":"ContainerStarted","Data":"84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358"} Oct 30 15:04:41.660761 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.660638 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" event={"ID":"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662","Type":"ContainerStarted","Data":"6a5c728e0ad66a7fd950aa775d47c0d53b53f2bd2c7994c346755ca8a09bde37"} Oct 30 15:04:41.660761 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.660706 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:41.704487 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.704439 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" podStartSLOduration=1.270159008 podStartE2EDuration="1.70442559s" totalImagesPullingTime="434.266582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:41.704135944 +0000 UTC m=+4511.504641390" watchObservedRunningTime="2026-10-30 15:04:41.70442559 +0000 UTC m=+4511.504931024" Oct 30 15:04:41.750017 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:41.749973 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" podStartSLOduration=1.255576192 podStartE2EDuration="1.749958863s" totalImagesPullingTime="494.382671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:41.749020578 +0000 UTC m=+4511.549526022" watchObservedRunningTime="2026-10-30 15:04:41.749958863 +0000 UTC m=+4511.550464301" Oct 30 15:04:47.671378 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:47.671343 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-2pczd" Oct 30 15:04:47.671892 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:47.671871 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:47.803794 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:47.803760 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q"] Oct 30 15:04:47.803972 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:47.803949 5824 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" podUID="5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" containerName="maas-api" containerID="cri-o://84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358" gracePeriod=30 Oct 30 15:04:48.055779 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.055754 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:48.156855 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.156819 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-openshift-service-ca\") pod \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " Oct 30 15:04:48.157056 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.156969 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-maas-api-tls\") pod \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " Oct 30 15:04:48.157056 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.157025 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-metrics-certs\") pod \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " Oct 30 15:04:48.157224 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.157078 5824 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-kube-api-access-cfrld\" (UniqueName: \"kubernetes.io/projected/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-kube-api-access-cfrld\") pod \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\" (UID: \"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662\") " Oct 30 15:04:48.157224 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.157156 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-openshift-service-ca" pod "5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" (UID: "5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:48.157434 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.157414 5824 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-openshift-service-ca\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:48.159292 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.159257 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-maas-api-tls" pod "5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" (UID: "5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:48.159420 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.159314 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-metrics-certs" pod "5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" (UID: "5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:48.159420 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.159349 5824 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-kube-api-access-cfrld" pod "5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" (UID: "5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662"). InnerVolumeSpecName "kube-api-access-cfrld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:48.258465 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.258423 5824 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-maas-api-tls\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:48.258465 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.258453 5824 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-metrics-certs\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:48.258465 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.258464 5824 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfrld\" (UniqueName: \"kubernetes.io/projected/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662-kube-api-access-cfrld\") on node \"ip-10-0-131-200.ec2.internal\" DevicePath \"\"" Oct 30 15:04:48.698373 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.698330 5824 generic.go:356] "Generic (PLEG): container finished" podID="5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" containerID="84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358" exitCode=0 Oct 30 15:04:48.698849 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.698383 5824 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" Oct 30 15:04:48.698849 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.698414 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" event={"ID":"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662","Type":"ContainerDied","Data":"84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358"} Oct 30 15:04:48.698849 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.698470 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q" event={"ID":"5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662","Type":"ContainerDied","Data":"6a5c728e0ad66a7fd950aa775d47c0d53b53f2bd2c7994c346755ca8a09bde37"} Oct 30 15:04:48.698849 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.698492 5824 scope.go:122] "RemoveContainer" containerID="84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358" Oct 30 15:04:48.707552 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.707525 5824 scope.go:122] "RemoveContainer" containerID="84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358" Oct 30 15:04:48.707837 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:04:48.707814 5824 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358\": container with ID starting with 84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358 not found: ID does not exist" containerID="84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358" Oct 30 15:04:48.707928 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.707848 5824 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358"} err="failed to get container status \"84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358\": rpc error: code = NotFound desc = could not find container \"84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358\": container with ID starting with 84b47488316a723c8cb673da731c551a511e36f5f6cffb45f654ac1ee89ae358 not found: ID does not exist" Oct 30 15:04:48.762743 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.762704 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q"] Oct 30 15:04:48.775993 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:48.775964 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-jdr4q"] Oct 30 15:04:50.315106 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:04:50.315060 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" path="/var/lib/kubelet/pods/5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662/volumes" Oct 30 15:05:03.640113 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:05:03.640078 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:05:33.537760 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:05:33.537724 5824 scope.go:122] "RemoveContainer" containerID="4745f99c433e1e62dc32a0c49eb71ddec271e25c75b79689e620f2628e827f60" Oct 30 15:05:33.640796 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:05:33.640752 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:05:50.034473 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:05:50.034437 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs"] Oct 30 15:05:50.047982 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:05:50.047954 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwrcs"] Oct 30 15:05:50.315849 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:05:50.315778 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30542cd7-a500-44d0-8fd1-6cbc316b9d28" path="/var/lib/kubelet/pods/30542cd7-a500-44d0-8fd1-6cbc316b9d28/volumes" Oct 30 15:06:03.640541 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:06:03.640500 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:06:05.032975 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:06:05.032927 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz"] Oct 30 15:06:05.046242 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:06:05.046213 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jl9mz"] Oct 30 15:06:06.315961 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:06:06.315922 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02600861-c39c-4219-95fc-c2ca52e0bd43" path="/var/lib/kubelet/pods/02600861-c39c-4219-95fc-c2ca52e0bd43/volumes" Oct 30 15:06:33.640568 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:06:33.640491 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:07:03.671212 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:07:03.664791 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:07:19.331943 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:07:19.331911 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cd6b76db-pbblr_3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e/authorino/0.log" Oct 30 15:07:33.568681 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:07:33.568649 5824 scope.go:122] "RemoveContainer" containerID="a1c89379351ee925f5e98f9327d70518c7cc65b0a1c38e92606fd94c44b40961" Oct 30 15:07:33.577340 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:07:33.577317 5824 scope.go:122] "RemoveContainer" containerID="50664dccfafaca5ce14ae5503bef7ba7c4de7cafc5804ad5f1e88f8f2ec95f8a" Oct 30 15:07:33.642914 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:07:33.642876 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:07:51.414793 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:07:51.414706 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-pvg7l_c5747106-030e-4cbf-aed2-ff89cf0e457e/manager/0.log" Oct 30 15:08:02.391749 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:02.391718 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-pnn8l_b81c6250-d13e-49ae-ab7e-1372263cd4b5/copy-manifests/0.log" Oct 30 15:08:02.427361 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:02.427324 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-pnn8l_b81c6250-d13e-49ae-ab7e-1372263cd4b5/manager/0.log" Oct 30 15:08:02.646946 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:02.646860 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-q2dfw_c048d8ea-ea88-4cbd-ac80-892f46baa62d/proxy/0.log" Oct 30 15:08:03.443165 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:03.443127 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-2pczd_df9b2fc0-487d-47e6-b5e4-de49add2c8ae/maas-api/0.log" Oct 30 15:08:03.642145 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:08:03.642108 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:08:04.042042 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:04.042011 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-31fef4f7-jvg8s_daf21a88-038d-44c1-9619-085949318f0c/revoke-keys/0.log" Oct 30 15:08:04.228473 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:04.228439 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-4kt7q_71a47e6d-43c2-4861-affa-73e5dae45cf0/revoke-keys/0.log" Oct 30 15:08:04.417954 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:04.417871 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-9p252_2d2e6988-4032-4495-8a42-c293bb0abbd0/revoke-keys/0.log" Oct 30 15:08:06.503949 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:06.503908 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cd6b76db-pbblr_3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e/authorino/0.log" Oct 30 15:08:06.983632 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:06.983603 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-gzmcs_d420195b-3cda-4417-88d0-752e1e4018ef/manager/0.log" Oct 30 15:08:07.840251 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:07.840217 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-chmkj_49d3309e-57a8-4f1a-92d9-e5521039d3f9/manager/0.log" Oct 30 15:08:08.994256 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:08.994222 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8996685d-dlgtt_690e8090-6456-4112-967f-adb1a133841d/kube-auth-proxy/0.log" Oct 30 15:08:09.593603 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:09.593570 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-cl2qx_6bc4102f-81bc-4831-bf65-fcd79c204148/istio-proxy/0.log" Oct 30 15:08:09.782474 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:09.782441 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-d7k5x_2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca/istio-proxy/0.log" Oct 30 15:08:10.562099 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:10.562059 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gqq8b_d5057b65-74d7-4c84-b500-ba697b16459f/istio-proxy/0.log" Oct 30 15:08:22.772465 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:22.772423 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-84kw9_c93f2a70-0763-42c9-b922-b1bec73d6a05/main/0.log" Oct 30 15:08:23.175530 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:23.175496 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-c5gqv_ad920ab1-dd5c-4167-b255-c68d6b9b8bc4/main/0.log" Oct 30 15:08:23.602772 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:23.602667 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2_d6a5964c-caae-4580-9c21-03a9c53f86b9/main/0.log" Oct 30 15:08:23.806704 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:23.806664 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz_7c410a7f-e3b9-485d-bbd4-2b3d83c2d297/main/0.log" Oct 30 15:08:24.814271 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:24.814225 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cd6b76db-pbblr_3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e/authorino/0.log" Oct 30 15:08:33.642475 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:08:33.642434 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:08:45.934952 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:45.934914 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-pvg7l_c5747106-030e-4cbf-aed2-ff89cf0e457e/manager/0.log" Oct 30 15:08:46.921265 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:46.921227 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-pnn8l_b81c6250-d13e-49ae-ab7e-1372263cd4b5/copy-manifests/0.log" Oct 30 15:08:46.957431 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:46.957399 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d55fbd5c9-pnn8l_b81c6250-d13e-49ae-ab7e-1372263cd4b5/manager/0.log" Oct 30 15:08:47.167946 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:47.167909 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-q2dfw_c048d8ea-ea88-4cbd-ac80-892f46baa62d/proxy/0.log" Oct 30 15:08:47.731148 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:47.731114 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-2pczd_df9b2fc0-487d-47e6-b5e4-de49add2c8ae/maas-api/0.log" Oct 30 15:08:48.362025 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:48.361992 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-31fef4f7-jvg8s_daf21a88-038d-44c1-9619-085949318f0c/revoke-keys/0.log" Oct 30 15:08:48.556620 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:48.556587 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-4kt7q_71a47e6d-43c2-4861-affa-73e5dae45cf0/revoke-keys/0.log" Oct 30 15:08:48.751544 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:48.751512 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-9p252_2d2e6988-4032-4495-8a42-c293bb0abbd0/revoke-keys/0.log" Oct 30 15:08:49.034259 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:49.034151 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252"] Oct 30 15:08:49.047516 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:49.047469 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-9p252"] Oct 30 15:08:50.315746 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:50.315709 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2e6988-4032-4495-8a42-c293bb0abbd0" path="/var/lib/kubelet/pods/2d2e6988-4032-4495-8a42-c293bb0abbd0/volumes" Oct 30 15:08:50.540150 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:50.540112 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cd6b76db-pbblr_3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e/authorino/0.log" Oct 30 15:08:51.001838 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:08:51.001799 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-gzmcs_d420195b-3cda-4417-88d0-752e1e4018ef/manager/0.log" Oct 30 15:09:01.920762 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:01.920714 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-chmkj_49d3309e-57a8-4f1a-92d9-e5521039d3f9/manager/0.log" Oct 30 15:09:03.642182 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:09:03.642135 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:09:09.034248 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:09.034205 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q"] Oct 30 15:09:09.047429 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:09.047390 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-4kt7q"] Oct 30 15:09:10.315168 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:10.315131 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a47e6d-43c2-4861-affa-73e5dae45cf0" path="/var/lib/kubelet/pods/71a47e6d-43c2-4861-affa-73e5dae45cf0/volumes" Oct 30 15:09:12.954159 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:12.954126 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8996685d-dlgtt_690e8090-6456-4112-967f-adb1a133841d/kube-auth-proxy/0.log" Oct 30 15:09:13.568527 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:13.568483 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-cl2qx_6bc4102f-81bc-4831-bf65-fcd79c204148/istio-proxy/0.log" Oct 30 15:09:13.776803 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:13.776768 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-d7k5x_2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca/istio-proxy/0.log" Oct 30 15:09:14.609859 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:14.609824 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gqq8b_d5057b65-74d7-4c84-b500-ba697b16459f/istio-proxy/0.log" Oct 30 15:09:26.599450 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:26.599412 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-84kw9_c93f2a70-0763-42c9-b922-b1bec73d6a05/main/0.log" Oct 30 15:09:27.019798 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:27.019764 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-c5gqv_ad920ab1-dd5c-4167-b255-c68d6b9b8bc4/main/0.log" Oct 30 15:09:27.431512 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:27.431483 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-xnbb2_d6a5964c-caae-4580-9c21-03a9c53f86b9/main/0.log" Oct 30 15:09:27.641860 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:27.641813 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-x4qxz_7c410a7f-e3b9-485d-bbd4-2b3d83c2d297/main/0.log" Oct 30 15:09:31.036629 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:31.036582 5824 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s"] Oct 30 15:09:31.049940 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:31.049904 5824 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fef4f7-jvg8s"] Oct 30 15:09:32.315504 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:09:32.315463 5824 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf21a88-038d-44c1-9619-085949318f0c" path="/var/lib/kubelet/pods/daf21a88-038d-44c1-9619-085949318f0c/volumes" Oct 30 15:09:33.642825 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:09:33.642783 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:10:03.643040 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:10:03.642992 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:10:33.614296 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:10:33.614258 5824 scope.go:122] "RemoveContainer" containerID="4c2ebac8704c9941ff5f7c01e4bc1f18f6f7e5aed6548963bb110ddd2fda75c7" Oct 30 15:10:33.624025 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:10:33.623994 5824 scope.go:122] "RemoveContainer" containerID="91ce3e299704ea4cca96c39cc8d947a2966e6bdaabc10c75c120cf7ead1ee62d" Oct 30 15:10:33.634125 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:10:33.634100 5824 scope.go:122] "RemoveContainer" containerID="473071bde90bfc4a140722dde2f0c7f81611091a9526783eb4f635c4dbe3c1c1" Oct 30 15:10:33.646767 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:10:33.646725 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:11:03.641799 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:11:03.641711 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:11:33.453819 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:33.453771 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pb5tm_1b37398e-d79b-4ced-a7b6-07409e2dbdcc/global-pull-secret-syncer/0.log" Oct 30 15:11:33.642857 ip-10-0-131-200 kubenswrapper[5824]: E1030 15:11:33.642817 5824 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6ltg7" containerName="extractor" Oct 30 15:11:34.404839 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:34.404790 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-stdsc_5cdb85a7-7e6c-4e60-bbac-dbf902150c60/konnectivity-agent/0.log" Oct 30 15:11:34.511310 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:34.511270 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-200.ec2.internal_bda9b1f2dbe238b1af8586cc49b37f56/haproxy/0.log" Oct 30 15:11:40.733809 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:40.733752 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cd6b76db-pbblr_3dcf125f-03ee-4aa9-bbc7-70c4fb1cc61e/authorino/0.log" Oct 30 15:11:40.971862 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:40.971826 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-gzmcs_d420195b-3cda-4417-88d0-752e1e4018ef/manager/0.log" Oct 30 15:11:41.546432 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:41.546374 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-chmkj_49d3309e-57a8-4f1a-92d9-e5521039d3f9/manager/0.log" Oct 30 15:11:42.177156 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.177112 5824 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd"] Oct 30 15:11:42.177662 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.177644 5824 cpu_manager.go:414] "containerMap: removing container" podUID="5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" containerName="maas-api" Oct 30 15:11:42.177714 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.177663 5824 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" containerName="maas-api" Oct 30 15:11:42.177803 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.177792 5824 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b48858d-e7c4-4cc9-aa2a-f1c07e6fc662" containerName="maas-api" Oct 30 15:11:42.181106 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.181079 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.198452 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.198412 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd"] Oct 30 15:11:42.210263 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.210229 5824 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-48dpl\"/\"default-dockercfg-dvdxx\"" Oct 30 15:11:42.220954 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.220922 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48dpl\"/\"openshift-service-ca.crt\"" Oct 30 15:11:42.220954 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.220951 5824 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48dpl\"/\"kube-root-ca.crt\"" Oct 30 15:11:42.280623 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.280552 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-proc\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.280623 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.280624 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475m8\" (UniqueName: \"kubernetes.io/projected/cf25549c-fa51-456b-9e65-834ecd875632-kube-api-access-475m8\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.280878 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.280803 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-lib-modules\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.280878 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.280846 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-podres\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.280955 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.280926 5824 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-sys\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382644 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382597 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-lib-modules\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382644 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382651 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-podres\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382676 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-sys\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382722 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-proc\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382743 5824 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475m8\" (UniqueName: \"kubernetes.io/projected/cf25549c-fa51-456b-9e65-834ecd875632-kube-api-access-475m8\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382748 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-lib-modules\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382786 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-podres\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382794 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-sys\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.382929 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.382799 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf25549c-fa51-456b-9e65-834ecd875632-proc\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.408918 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.408875 5824 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-475m8\" (UniqueName: \"kubernetes.io/projected/cf25549c-fa51-456b-9e65-834ecd875632-kube-api-access-475m8\") pod \"perf-node-gather-daemonset-f5ljd\" (UID: \"cf25549c-fa51-456b-9e65-834ecd875632\") " pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.495079 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.494974 5824 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:42.824732 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:42.824700 5824 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd"] Oct 30 15:11:42.826574 ip-10-0-131-200 kubenswrapper[5824]: W1030 15:11:42.826540 5824 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcf25549c_fa51_456b_9e65_834ecd875632.slice/crio-4791a154c2f2288e9639f08026ab185881f0435ed87c57ae379154a1171d485b WatchSource:0}: Error finding container 4791a154c2f2288e9639f08026ab185881f0435ed87c57ae379154a1171d485b: Status 404 returned error can't find the container with id 4791a154c2f2288e9639f08026ab185881f0435ed87c57ae379154a1171d485b Oct 30 15:11:43.045655 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.045621 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1406a3ec-1760-435e-b89a-48ffdc425a94/alertmanager/0.log" Oct 30 15:11:43.167214 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.167115 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1406a3ec-1760-435e-b89a-48ffdc425a94/config-reloader/0.log" Oct 30 15:11:43.286693 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.286643 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1406a3ec-1760-435e-b89a-48ffdc425a94/kube-rbac-proxy-web/0.log" Oct 30 15:11:43.405254 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.405172 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1406a3ec-1760-435e-b89a-48ffdc425a94/kube-rbac-proxy/0.log" Oct 30 15:11:43.521351 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.521321 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1406a3ec-1760-435e-b89a-48ffdc425a94/kube-rbac-proxy-metric/0.log" Oct 30 15:11:43.639921 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.639886 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1406a3ec-1760-435e-b89a-48ffdc425a94/prom-label-proxy/0.log" Oct 30 15:11:43.758438 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.758405 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1406a3ec-1760-435e-b89a-48ffdc425a94/init-config-reloader/0.log" Oct 30 15:11:43.794786 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.794689 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" event={"ID":"cf25549c-fa51-456b-9e65-834ecd875632","Type":"ContainerStarted","Data":"368f4525fc4163cb84de73423a46d0fefce2e0b4f1653b0373e32a6124f7672b"} Oct 30 15:11:43.794786 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.794748 5824 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" event={"ID":"cf25549c-fa51-456b-9e65-834ecd875632","Type":"ContainerStarted","Data":"4791a154c2f2288e9639f08026ab185881f0435ed87c57ae379154a1171d485b"} Oct 30 15:11:43.795027 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.794829 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:43.844751 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:43.844689 5824 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" podStartSLOduration=1.844673531 podStartE2EDuration="1.844673531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:43.842546578 +0000 UTC m=+4933.643052022" watchObservedRunningTime="2026-10-30 15:11:43.844673531 +0000 UTC m=+4933.645178968" Oct 30 15:11:44.003011 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:44.002975 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cz5xj_80844f10-0b8f-4f5e-bbba-4a93bb0a8626/kube-state-metrics/0.log" Oct 30 15:11:44.116808 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:44.116719 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cz5xj_80844f10-0b8f-4f5e-bbba-4a93bb0a8626/kube-rbac-proxy-main/0.log" Oct 30 15:11:44.237151 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:44.237119 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cz5xj_80844f10-0b8f-4f5e-bbba-4a93bb0a8626/kube-rbac-proxy-self/0.log" Oct 30 15:11:44.361333 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:44.361300 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76d44865b4-8hgxr_9d35bccc-a939-4913-8903-63b51885cc9a/metrics-server/0.log" Oct 30 15:11:45.392264 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:45.392223 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsrlt_d413c8fe-be32-46db-9326-5cfb42fc987b/node-exporter/0.log" Oct 30 15:11:45.508614 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:45.508581 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsrlt_d413c8fe-be32-46db-9326-5cfb42fc987b/kube-rbac-proxy/0.log" Oct 30 15:11:45.625532 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:45.625506 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsrlt_d413c8fe-be32-46db-9326-5cfb42fc987b/init-textfile/0.log" Oct 30 15:11:46.147656 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:46.147622 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cac7de6a-e8ba-4f54-b53d-a3e16b607ac7/prometheus/0.log" Oct 30 15:11:46.268941 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:46.268886 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cac7de6a-e8ba-4f54-b53d-a3e16b607ac7/config-reloader/0.log" Oct 30 15:11:46.413853 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:46.413767 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cac7de6a-e8ba-4f54-b53d-a3e16b607ac7/thanos-sidecar/0.log" Oct 30 15:11:46.546105 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:46.546076 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cac7de6a-e8ba-4f54-b53d-a3e16b607ac7/kube-rbac-proxy-web/0.log" Oct 30 15:11:46.689182 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:46.689153 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cac7de6a-e8ba-4f54-b53d-a3e16b607ac7/kube-rbac-proxy/0.log" Oct 30 15:11:46.825658 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:46.825621 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cac7de6a-e8ba-4f54-b53d-a3e16b607ac7/kube-rbac-proxy-thanos/0.log" Oct 30 15:11:46.946384 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:46.946275 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cac7de6a-e8ba-4f54-b53d-a3e16b607ac7/init-config-reloader/0.log" Oct 30 15:11:47.082380 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:47.082346 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4dblz_45341aa5-a0c9-4241-ae17-fa2f1b223f74/prometheus-operator/0.log" Oct 30 15:11:47.201108 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:47.201026 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-4dblz_45341aa5-a0c9-4241-ae17-fa2f1b223f74/kube-rbac-proxy/0.log" Oct 30 15:11:47.941887 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:47.941854 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddbd5f6b-7kc79_99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/thanos-query/0.log" Oct 30 15:11:48.065585 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:48.065553 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddbd5f6b-7kc79_99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/kube-rbac-proxy-web/0.log" Oct 30 15:11:48.184260 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:48.184229 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddbd5f6b-7kc79_99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/kube-rbac-proxy/0.log" Oct 30 15:11:48.303438 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:48.303351 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddbd5f6b-7kc79_99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/prom-label-proxy/0.log" Oct 30 15:11:48.421966 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:48.421937 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddbd5f6b-7kc79_99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/kube-rbac-proxy-rules/0.log" Oct 30 15:11:48.546650 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:48.546620 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddbd5f6b-7kc79_99134c6b-8fa1-443d-ac56-a69a2bfcc9cb/kube-rbac-proxy-metrics/0.log" Oct 30 15:11:49.814659 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:49.814625 5824 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-48dpl/perf-node-gather-daemonset-f5ljd" Oct 30 15:11:55.944830 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:55.944795 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7n7tm_c80b9b41-b573-4246-b0ca-fd5e7297e6f0/dns/0.log" Oct 30 15:11:56.065123 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:56.065082 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7n7tm_c80b9b41-b573-4246-b0ca-fd5e7297e6f0/kube-rbac-proxy/0.log" Oct 30 15:11:56.707589 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:56.707558 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p7256_72a2c31f-6357-4b42-be58-9672ec8dcecb/dns-node-resolver/0.log" Oct 30 15:11:57.956813 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:57.956762 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8996685d-dlgtt_690e8090-6456-4112-967f-adb1a133841d/kube-auth-proxy/0.log" Oct 30 15:11:58.145781 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:58.145737 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pscjw_7e9025a0-9daa-426a-b18d-3a04a154b693/node-ca/0.log" Oct 30 15:11:58.162018 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:58.161983 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-cl2qx_6bc4102f-81bc-4831-bf65-fcd79c204148/istio-proxy/0.log" Oct 30 15:11:58.228951 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:58.228871 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-d7k5x_2dbddc3f-7abd-4aa5-a42b-f018fe7e65ca/istio-proxy/0.log" Oct 30 15:11:58.501389 ip-10-0-131-200 kubenswrapper[5824]: I1030 15:11:58.501303 5824 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gqq8b_d5057b65-74d7-4c84-b500-ba697b16459f/istio-proxy/0.log"